Jul 10 20:19:05.472269 ip-10-0-141-240 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 10 20:19:05.472280 ip-10-0-141-240 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 10 20:19:05.472287 ip-10-0-141-240 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 10 20:19:05.472495 ip-10-0-141-240 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 10 20:19:15.512200 ip-10-0-141-240 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 10 20:19:15.512218 ip-10-0-141-240 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bbed2ef5a190498ba1a05acd33a81d88 --
Jul 10 20:21:14.849382 ip-10-0-141-240 systemd[1]: Starting Kubernetes Kubelet...
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.322588    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326149    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326161    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326166    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326169    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326172    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326176    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326179    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:15.407074 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326181    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326184    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326187    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326195    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326199    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326202    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326204    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326207    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326211    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326214    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326218    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326222    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326225    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326227    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326230    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326234    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326238    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326241    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326243    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:15.414060 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326246    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326250    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326252    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326255    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326258    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326260    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326263    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326265    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326268    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326271    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326273    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326276    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326279    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326281    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326284    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326286    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326289    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326292    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326295    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326297    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:15.414705 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326299    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326303    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326305    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326308    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326310    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326312    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326315    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326317    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326320    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326322    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326325    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326327    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326330    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326334    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326337    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326339    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326342    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326344    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326347    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326350    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:15.415357 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326352    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326355    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326357    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326360    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326363    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326366    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326370    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326373    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326376    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326379    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326381    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326384    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326386    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326389    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326391    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326393    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326396    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326399    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326401    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326405    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:15.416011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326407    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326410    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326412    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326415    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326418    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326421    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326424    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326426    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326429    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326431    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326436    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.326439    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327141    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327148    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327151    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327154    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327156    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327160    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327163    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:15.416661 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327166    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327168    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327171    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327174    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327176    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327179    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327181    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327184    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327186    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327189    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327191    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327195    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327197    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327200    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327202    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327205    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327208    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327210    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327213    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327215    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:15.417298 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327218    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327220    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327223    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327225    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327229    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327231    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327234    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327236    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327239    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327241    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327244    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327247    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327249    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327252    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327254    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327258    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327263    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327266    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327269    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327271    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:15.417957 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327274    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327277    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327279    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327282    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327285    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327287    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327290    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327292    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327295    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327298    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327300    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327303    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327305    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327308    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327310    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327313    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327316    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327319    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327322    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:15.418604 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327324    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327327    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327330    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327332    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327335    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327338    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327340    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327343    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327346    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327348    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327351    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327353    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327356    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327359    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327361    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327363    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327366    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327370    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327373    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327375    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:15.419254 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327379    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327382    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327386    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327390    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327393    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327397    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327400    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327404    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327407    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327410    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327413    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327415    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327418    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327796    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327801    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327804    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327807    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327812    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327815    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:15.419931 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327818    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327821    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327823    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327826    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327829    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327831    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327834    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327837    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327842    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327845    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327848    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327850    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327853    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327855    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327858    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327861    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327865    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327868    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327871    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327874    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:15.420584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327877    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327879    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327882    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327886    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327888    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327891    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327894    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327896    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327899    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327902    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327904    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327910    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327913    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327917    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327920    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327922    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327925    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327928    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327930    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327933    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:15.421441 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327940    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327942    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327946    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327948    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327951    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327954    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327956    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327959    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327961    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327964    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327966    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327969    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327971    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327974    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327976    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327979    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327981    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327984    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327986    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327989    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:15.422089 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327991    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327994    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327996    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.327999    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328011    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328014    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328016    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328019    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328022    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328024    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328027    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328031    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328037    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328041    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328045    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328048    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328051    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328054    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328057    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:15.422729 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328060    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328063    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328066    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328069    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328072    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328075    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328078    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328082    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328085    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328087    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328090    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328093    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328095    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328098    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328208    2564 flags.go:64] FLAG: --address="0.0.0.0"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328219    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328229    2564 flags.go:64] FLAG: --anonymous-auth="true"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328234    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328238    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328242    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328249    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 10 20:21:15.423249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328253    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328256    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328259    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328263    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328271    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328274    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328277    2564 flags.go:64] FLAG: --cgroup-root=""
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328280    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328283    2564 flags.go:64] FLAG: --client-ca-file=""
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328286    2564 flags.go:64] FLAG: --cloud-provider="external"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328289    2564 flags.go:64] FLAG: --cluster-dns="[]"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328293    2564 flags.go:64] FLAG: --cluster-domain=""
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328296    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328299    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328302    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328305    2564 flags.go:64] FLAG: --container-log-max-files="5"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328309    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328312    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328315    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328318    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328322    2564 flags.go:64] FLAG: --contention-profiling="false"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328324    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328327    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328330    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 10 20:21:15.423806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328333    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328337    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328340    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328343    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328347    2564 flags.go:64] FLAG: --enable-load-reader="false"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328350    2564 flags.go:64] FLAG: --enable-server="true"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328353    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328358    2564 flags.go:64] FLAG: --event-burst="100"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328361    2564 flags.go:64] FLAG: --event-qps="50"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328364    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328367    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328370    2564 flags.go:64] FLAG: --eviction-hard=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328376    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328379    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328382    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328385    2564 flags.go:64] FLAG: --eviction-soft=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328388    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328391    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328394    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328397    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328400    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328403    2564 flags.go:64] FLAG: --fail-swap-on="true"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328406    2564 flags.go:64] FLAG: --feature-gates=""
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328409    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328412    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 10 20:21:15.424438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328415    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328419    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328422    2564 flags.go:64] FLAG: --healthz-port="10248"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328425    2564 flags.go:64] FLAG: --help="false"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328428    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328430    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328433    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328436    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328440    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328443    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328446    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328449    2564 flags.go:64] FLAG: --image-service-endpoint=""
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328452    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328454    2564 flags.go:64] FLAG: --kube-api-burst="100"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328458    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328461    2564 flags.go:64] FLAG: --kube-api-qps="50"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328464    2564 flags.go:64] FLAG: --kube-reserved=""
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328467    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328469    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328475    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328478    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328481    2564 flags.go:64] FLAG: --lock-file=""
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328484    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328487    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 10 20:21:15.425182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328489    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328495    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328498    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328500    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328503    2564 flags.go:64] FLAG: --logging-format="text"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328506    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328510    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328513    2564 flags.go:64] FLAG: --manifest-url=""
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328516    2564 flags.go:64] FLAG: --manifest-url-header=""
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328520    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328523    2564 flags.go:64] FLAG: --max-open-files="1000000"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328527    2564 flags.go:64] FLAG: --max-pods="110"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328530    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328533    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328536    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328539    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328542    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328545    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328548    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328556    2564 flags.go:64] FLAG: --node-status-max-images="50"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328559    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328562    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328565    2564 flags.go:64] FLAG: --pod-cidr=""
Jul 10 20:21:15.495598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328568    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 10 20:21:15.429556 ip-10-0-141-240 systemd[1]: Started Kubernetes Kubelet.
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328574    2564 flags.go:64] FLAG: --pod-manifest-path=""
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328577    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328580    2564 flags.go:64] FLAG: --pods-per-core="0"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328585    2564 flags.go:64] FLAG: --port="10250"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328590    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328593    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-012567af0edd645e1"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328596    2564 flags.go:64] FLAG: --qos-reserved=""
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328599    2564 flags.go:64] FLAG: --read-only-port="10255"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328602    2564 flags.go:64] FLAG: --register-node="true"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328605    2564 flags.go:64] FLAG: --register-with-taints=""
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328609    2564 flags.go:64] FLAG: --registry-burst="10"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328612    2564 flags.go:64] FLAG: --registry-qps="5"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328614    2564 flags.go:64] FLAG: --reserved-cpus=""
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328617    2564 flags.go:64] FLAG: --reserved-memory=""
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328621    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328624    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328627    2564 flags.go:64] FLAG: --rotate-certificates="false"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328630    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328633    2564 flags.go:64] FLAG: --runonce="false"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328636    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328639    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328642    2564 flags.go:64] FLAG: --seccomp-default="false"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328645    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328648    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328650    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 10 20:21:15.496645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328653    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328657    2564 flags.go:64] FLAG: --storage-driver-password="root"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328660    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328662    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328665    2564 flags.go:64] FLAG: --storage-driver-user="root"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328668    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328671    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328674    2564 flags.go:64] FLAG: --system-cgroups=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328677    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328683    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328687    2564 flags.go:64] FLAG: --tls-cert-file=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328691    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328696    2564 flags.go:64] FLAG: --tls-min-version=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328698    2564 flags.go:64] FLAG: --tls-private-key-file=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328701    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328705    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328708    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328711    2564 flags.go:64] FLAG: --v="2"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328715    2564 flags.go:64] FLAG: --version="false"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328719    2564 flags.go:64] FLAG: --vmodule=""
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328723    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.328726    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328858    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328861    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328864    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:16.636449 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328867    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328871    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328874    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328877    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328880    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328883    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328886    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328888    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328891    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328893    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328896    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328899    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328901    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328904    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328906    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328909    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328912    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328916    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328920    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:16.637858 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328922    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328925    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328927    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328930    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328933    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328935    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328938    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328940    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328943    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328946    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328948    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328951    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328953    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328956    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328958    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328961    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328964    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328966    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328969    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:16.639627 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328972    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328976    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328978    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328981    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328983    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328986    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328988    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328991    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328993    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328996    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.328999    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329003    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329007    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329009    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329012    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329014    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329017    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329020    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329023    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329025    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:16.640750 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329028    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329030    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329033    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329035    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329038    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329041    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329043    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329046    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329048    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329051    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329055    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329058    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329061    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329064    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329067    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329069    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329072    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329074    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329078    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:16.641678 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329082    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329085    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329087    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329090    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329095    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329099    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329101    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329104    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329107    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329123    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329126    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329128    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329131    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329133    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329136    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329139    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329141    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329144    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:16.757758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.329146    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.329151    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.335698    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.335714    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.335727    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.335736    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335791    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335796    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335799    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335802    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335805    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335808    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335810    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335813    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:16.758556 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335816    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335818    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335821    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335823    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335828    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335832    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335835    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335837    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335840    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335843    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335845    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335848    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335851    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335853    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335856    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335858    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335861    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335864    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335867    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335870    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:16.759584 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335874    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335877    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335880    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335883    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335886    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335889    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335891    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335894    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335898    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335900    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335903    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335905    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335911    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335914    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335916    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335919    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335921    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335924    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335927    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335930    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:16.760350 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335933    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335936    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335938    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335941    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335943    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335946    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335948    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335951    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335954    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335957    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335959    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335962    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335965    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335969    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335975    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335979    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335981    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335984    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335987    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:16.761149 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335989    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335994    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335996    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.335999    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336001    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336005    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336008    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336011    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336014    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336016    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336019    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336022    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336024    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336027    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336029    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336032    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336035    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336038    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336040    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336043    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:16.761758 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336045    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336048    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336050    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336053    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336056    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336059    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336061    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336064    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336067    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336069    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336072    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336074    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.336079    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336210    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336215    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 10 20:21:16.762742 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336218    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336221    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336224    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336228    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336231    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336235    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336239    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336242    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336245    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336248    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336251    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336254    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336257    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336259    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336262    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336265    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336268    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336270    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336273    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 10 20:21:16.763364 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336276    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336278    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336281    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336283    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336286    2564 feature_gate.go:328] unrecognized feature gate: Example2
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336288    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336291    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336294    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336296    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336299    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336301    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336305    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336307    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336310    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336312    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336315    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336319    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336322    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336325    2564 feature_gate.go:328] unrecognized feature gate: Example
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336327    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 10 20:21:16.763970 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336329    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336332    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336334    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336337    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336339    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336342    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336344    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336347    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336350    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336352    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336355    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336357    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336360    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336363    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336365    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336367    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336370    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336373    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336375    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336378    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 10 20:21:16.764570 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336380    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336383    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336385    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336389    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336392    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336394    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336397    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336399    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336403    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336405    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336408    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336411    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336413    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336416    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336418    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336421    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336423    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336426    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336429    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336431    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 10 20:21:16.765139 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336434    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336437    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336441    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336444    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336447    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336450    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336452    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336455    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336457    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336460    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336463    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336465    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336467    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336470    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336473    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336477    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336480    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 10 20:21:16.765707 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:15.336484    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.336489    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.336587    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.347151    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.348407    2564 server.go:1019] "Starting client certificate rotation"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.348508    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.348548    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.350733    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.353328    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.367386    2564 log.go:25] "Validated CRI v1 runtime API"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.369965    2564 log.go:25] "Validated CRI v1 image API"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.370903    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 10 20:21:16.766267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.374331    2564 fs.go:135] Filesystem UUIDs: map[57ca20a1-532b-41a1-8535-aafdf566b46f:/dev/nvme0n1p3 5ff77e51-8e5b-4800-9097-a195ce5f3ffa:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 10 20:21:16.766661 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.374344    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 10 20:21:16.766661 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.376625    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.379895    2564 manager.go:216] Machine: {Timestamp:2026-07-10 20:21:15.377830679 +0000 UTC m=+0.414511430 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086935 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2611c8248f26ab3d2ac2ca360bd8bf SystemUUID:ec2611c8-248f-26ab-3d2a-c2ca360bd8bf BootID:bbed2ef5-a190-498b-a1a0-5acd33a81d88 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:81:95:0b:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:81:95:0b:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:8e:72:e4:71:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.380007    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.380082    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.383051    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.383085    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-240.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.383931    2564 topology_manager.go:138] "Creating topology manager with none policy"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.383951    2564 container_manager_linux.go:309] "Creating device plugin manager"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.383961    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.384720    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.385454    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388197    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388313    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388342    2564 kubelet.go:486] "Attempting to sync node with API server"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388367    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388381    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388390    2564 kubelet.go:398] "Adding apiserver pod source"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388401    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 10 20:21:16.766836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388727    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.388744    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.396977    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:15.407086    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:15.407106    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.421579    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9rrj"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.423583    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.423612    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424216    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424232    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424242    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424251    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424259    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424264    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424360    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424365    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424372    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424377    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424386    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.424393    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.425091    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.425097    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.428303    2564 server.go:1268] "Started kubelet"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.428394    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.428556    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.428628    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.428897    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w9rrj"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.430080    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.430755    2564 server.go:310] "Adding debug handlers to kubelet server"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.435226    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.435237    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:15.435327    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.435800    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.435816    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.435850    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.436149    2564 reconstruct.go:97] "Volume reconstruction finished"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.436158    2564 reconciler.go:29] "Reconciler: start to sync state"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444126    2564 factory.go:55] Registering systemd factory
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444142    2564 factory.go:223] Registration of the systemd container factory successfully
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444807    2564 factory.go:153] Registering CRI-O factory
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444823    2564 factory.go:223] Registration of the crio container factory successfully
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444878    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444898    2564 factory.go:103] Registering Raw factory
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.444912    2564 manager.go:1196] Started watching for new ooms in manager
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:15.445809    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.446045    2564 manager.go:349] Starting recovery of all containers
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.454188    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:15.455019    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-240.ec2.internal\" not found" node="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.767437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.455430    2564 manager.go:354] Recovery completed
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.459132    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-240.ec2.internal" not found
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.465328    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467000    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467026    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467035    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467580    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467590    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467596    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.467605    2564 state_mem.go:36] "Initialized new in-memory state store"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468236    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468283    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468298    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468696    2564 policy_none.go:49] "None policy: Start"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468714    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.468727    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.469619    2564 policy_none.go:47] "Start"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:15.474978    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-240.ec2.internal" not found
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661264    2564 manager.go:337] "Starting Device Plugin manager"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.661300    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661310    2564 server.go:87] "Starting device plugin registration server"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661589    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661602    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661700    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661799    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.661809    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.662348    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.662405    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.668968    2564 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-141-240.ec2.internal" not found
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.762707    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.763695    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.763727    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.763744    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.768956 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.763772    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.773517    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.773537    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-240.ec2.internal\": node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.776870    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.778111    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.778172    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.778197    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.778235    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.780330    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:16.863393 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.794965    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.879331    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"]
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.879410    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.880327    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.880355    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.880364    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884229    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884509    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884910    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884932    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.884942    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.885139    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.885159    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.885173    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.887119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.887141    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.887794    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientMemory"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.887812    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasNoDiskPressure"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.887827    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeHasSufficientPID"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.895234    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.918086    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-240.ec2.internal\" not found" node="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.922439    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-240.ec2.internal\" not found" node="ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.974624    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/869786661d939b62b49e72c2e18cdc59-config\") pod \"kube-apiserver-proxy-ip-10-0-141-240.ec2.internal\" (UID: \"869786661d939b62b49e72c2e18cdc59\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.974645    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:16.992252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:16.974662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:16.995857    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/869786661d939b62b49e72c2e18cdc59-config\") pod \"kube-apiserver-proxy-ip-10-0-141-240.ec2.internal\" (UID: \"869786661d939b62b49e72c2e18cdc59\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a73838634c65a1c0797837ab62c2d67-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal\" (UID: \"0a73838634c65a1c0797837ab62c2d67\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.075572    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/869786661d939b62b49e72c2e18cdc59-config\") pod \"kube-apiserver-proxy-ip-10-0-141-240.ec2.internal\" (UID: \"869786661d939b62b49e72c2e18cdc59\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.139440 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.096468    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.260516 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.196897    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.260516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.220057    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.260516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.224685    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.418629 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.297475    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.418629 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.397916    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.528927 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.484423    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 10 20:21:17.528927 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.484579    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 20:21:17.528927 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.484584    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 10 20:21:17.528927 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.498695    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.651353 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.569991    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 20:16:16 +0000 UTC" deadline="2028-03-08 05:17:25.280710187 +0000 UTC"
Jul 10 20:21:17.651353 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.570013    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14552h56m7.71069986s"
Jul 10 20:21:17.651353 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:17.599454    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-240.ec2.internal\" not found"
Jul 10 20:21:17.651353 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.602259    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:17.796502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.671804    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.796502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.682851    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 20:21:17.796502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.684469    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal"
Jul 10 20:21:17.796502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.697527    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 10 20:21:17.796502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:17.790301    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:18.057722 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.057693    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869786661d939b62b49e72c2e18cdc59.slice/crio-bcb8a1d9bf53e13bef12e1bde7f7f1eb64bf5c9fa1d56c894c3bccda6cbad2c5 WatchSource:0}: Error finding container bcb8a1d9bf53e13bef12e1bde7f7f1eb64bf5c9fa1d56c894c3bccda6cbad2c5: Status 404 returned error can't find the container with id bcb8a1d9bf53e13bef12e1bde7f7f1eb64bf5c9fa1d56c894c3bccda6cbad2c5
Jul 10 20:21:18.058754 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.058730    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a73838634c65a1c0797837ab62c2d67.slice/crio-5e308723e8b716710bb18680d6937ffa724c48197e216ad85fcd28acf45b48f9 WatchSource:0}: Error finding container 5e308723e8b716710bb18680d6937ffa724c48197e216ad85fcd28acf45b48f9: Status 404 returned error can't find the container with id 5e308723e8b716710bb18680d6937ffa724c48197e216ad85fcd28acf45b48f9
Jul 10 20:21:18.062475 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.062459    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:21:18.525906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.525874    2564 apiserver.go:52] "Watching apiserver"
Jul 10 20:21:18.534349 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.534327    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 10 20:21:18.534737 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.534717    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hjqwh","openshift-multus/multus-4cnx8","openshift-network-diagnostics/network-check-target-ml7ds","openshift-network-operator/iptables-alerter-r6cd4","kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal","openshift-cluster-node-tuning-operator/tuned-zktcq","openshift-image-registry/node-ca-lnq7j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal","openshift-multus/multus-additional-cni-plugins-6ww8b","openshift-multus/network-metrics-daemon-cfp74","openshift-ovn-kubernetes/ovnkube-node-fqfdp","kube-system/global-pull-secret-syncer-gqc7d","kube-system/konnectivity-agent-r246c","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"]
Jul 10 20:21:18.538362 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.538340    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.538451 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.538428    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:18.540714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.540692    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.542850 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.542833    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:18.542952 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.542924    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543553    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543564    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mskjc\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543640    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543654    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543678    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543641    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.543824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.543641    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 10 20:21:18.545078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.545061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.547224 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.547207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.547526 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.547453    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.547826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.547801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 10 20:21:18.547916 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.547847    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hzbp7\""
Jul 10 20:21:18.547977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.547886    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.549773 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.549733    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.549773 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.549755    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 10 20:21:18.549948 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.549865    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.549948 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.549899    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.550064 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.549951    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-grs2f\""
Jul 10 20:21:18.552283 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.552259    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.552416 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.552371    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jr6h6\""
Jul 10 20:21:18.552514 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.552503    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.552600 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.552580    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.554555 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.554536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.554724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.554706    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-psrr5\""
Jul 10 20:21:18.555108 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.555090    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 10 20:21:18.555190 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.555107    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 10 20:21:18.555249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.555218    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.555249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.555241    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.555469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.555450    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 10 20:21:18.557012 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.556976    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.557413 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.557397    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 10 20:21:18.557659 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.557477    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d62gr\""
Jul 10 20:21:18.560648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.560215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dvpf\""
Jul 10 20:21:18.560648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.560273    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.560648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.560482    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.560837 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.560747    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.563614 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.563591    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 10 20:21:18.563701 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.563646    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mc449\""
Jul 10 20:21:18.563701 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.563664    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 10 20:21:18.564465 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.564451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.564681 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.564654    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:18.567414 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.567396    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.570088 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.570067    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 10 20:21:18.570706 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.570468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6snw\""
Jul 10 20:21:18.570706 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.570535    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 10 20:21:18.570706 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.570584    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 10 20:21:18.572381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.572157    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 10 20:21:18.573383 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.573365    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 10 20:21:18.582670 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.582652    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 10 20:21:18.584932 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.584909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-tmp\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.585017 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.584946    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-registration-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.585017 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.584971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-sys-fs\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.585132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585036    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwvbg\" (UniqueName: \"kubernetes.io/projected/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kube-api-access-cwvbg\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.585132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d0e425-31ca-4ab8-bd99-8534f583d901-host\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.585132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-hostroot\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-multus-certs\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585155    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsh2j\" (UniqueName: \"kubernetes.io/projected/e5997655-1f2d-49b4-b556-a1423236c106-kube-api-access-nsh2j\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-var-lib-kubelet\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-var-lib-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.585272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585255    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-cni-binary-copy\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-kubelet-config\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-etc-selinux\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585379    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-conf-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5997655-1f2d-49b4-b556-a1423236c106-iptables-alerter-script\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvxw\" (UniqueName: \"kubernetes.io/projected/b5d0e425-31ca-4ab8-bd99-8534f583d901-kube-api-access-fdvxw\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-bin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-kubelet\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585557    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-sys\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.585583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-netns\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-script-lib\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585632    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw6v\" (UniqueName: \"kubernetes.io/projected/395ceee6-e47f-4442-8897-6b4a0075f68b-kube-api-access-fpw6v\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-system-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395ceee6-e47f-4442-8897-6b4a0075f68b-hosts-file\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-config\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-conf\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-os-release\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-log-socket\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.585971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysconfig\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-cnibin\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.586051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586055    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395ceee6-e47f-4442-8897-6b4a0075f68b-tmp-dir\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-systemd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-device-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5d0e425-31ca-4ab8-bd99-8534f583d901-serviceca\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586127    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-cnibin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-kubernetes\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586169    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-node-log\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586182    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-os-release\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-k8s-cni-cncf-io\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-modprobe-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6681f43-e63d-495d-9870-62d20624cac7-konnectivity-ca\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-etc-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-netd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586279    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22863a35-4dda-44cb-ae0a-b922feaed80b-ovn-node-metrics-cert\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.586805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzt9\" (UniqueName: \"kubernetes.io/projected/d8317e36-4f62-40a7-91cd-0b523511bbb2-kube-api-access-zfzt9\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/74670a37-c1b7-4727-9de0-8281ca2176c1-kube-api-access-c2vz2\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-systemd-units\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586426    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-socket-dir-parent\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586451    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-etc-kubernetes\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-dbus\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-lib-modules\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586556    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-netns\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-multus\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586633    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-daemon-config\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgrh\" (UniqueName: \"kubernetes.io/projected/22863a35-4dda-44cb-ae0a-b922feaed80b-kube-api-access-wtgrh\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586673    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-systemd\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8nr\" (UniqueName: \"kubernetes.io/projected/2b72c49e-6395-4d9f-9495-d52457c4680c-kube-api-access-6n8nr\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.587363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-slash\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-ovn\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-bin\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586852    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-env-overrides\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-run\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-host\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdkq\" (UniqueName: \"kubernetes.io/projected/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-kube-api-access-4fdkq\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.586996    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6681f43-e63d-495d-9870-62d20624cac7-agent-certs\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.587018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-kubelet\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.587041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.587064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-socket-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.587087    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-tuned\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.588076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.587108    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5997655-1f2d-49b4-b556-a1423236c106-host-slash\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.603623 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.603605    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kgr6v"
Jul 10 20:21:18.611213 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.611195    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kgr6v"
Jul 10 20:21:18.688226 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-netns\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688281    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-multus\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-daemon-config\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgrh\" (UniqueName: \"kubernetes.io/projected/22863a35-4dda-44cb-ae0a-b922feaed80b-kube-api-access-wtgrh\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688341    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-systemd\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8nr\" (UniqueName: \"kubernetes.io/projected/2b72c49e-6395-4d9f-9495-d52457c4680c-kube-api-access-6n8nr\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-slash\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-ovn\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-bin\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-env-overrides\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-netns\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-multus\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688927    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.688965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-run\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.688990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-host\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdkq\" (UniqueName: \"kubernetes.io/projected/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-kube-api-access-4fdkq\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6681f43-e63d-495d-9870-62d20624cac7-agent-certs\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-kubelet\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-socket-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689112    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-daemon-config\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-tuned\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5997655-1f2d-49b4-b556-a1423236c106-host-slash\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-tmp\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-registration-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-sys-fs\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689219    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwvbg\" (UniqueName: \"kubernetes.io/projected/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kube-api-access-cwvbg\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d0e425-31ca-4ab8-bd99-8534f583d901-host\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-hostroot\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.689826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-multus-certs\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsh2j\" (UniqueName: \"kubernetes.io/projected/e5997655-1f2d-49b4-b556-a1423236c106-kube-api-access-nsh2j\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-var-lib-kubelet\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-var-lib-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-systemd\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-cni-binary-copy\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-run\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689530    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-kubelet-config\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-host\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-etc-selinux\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-conf-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689639    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5997655-1f2d-49b4-b556-a1423236c106-iptables-alerter-script\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvxw\" (UniqueName: \"kubernetes.io/projected/b5d0e425-31ca-4ab8-bd99-8534f583d901-kube-api-access-fdvxw\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-bin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.690619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689718    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-kubelet\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-sys\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-netns\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-script-lib\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-system-cni-dir\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689818    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw6v\" (UniqueName: \"kubernetes.io/projected/395ceee6-e47f-4442-8897-6b4a0075f68b-kube-api-access-fpw6v\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-system-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395ceee6-e47f-4442-8897-6b4a0075f68b-hosts-file\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-config\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.689976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-conf\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-os-release\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690083    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-log-socket\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.691454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-kubelet\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysconfig\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-slash\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690196    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-kubelet\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-sys\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-bin\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-socket-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e5997655-1f2d-49b4-b556-a1423236c106-host-slash\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-env-overrides\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-registration-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-sys-fs\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690853    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d0e425-31ca-4ab8-bd99-8534f583d901-host\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-hostroot\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-multus-certs\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-script-lib\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8317e36-4f62-40a7-91cd-0b523511bbb2-cni-binary-copy\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690994    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-kubelet-config\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.692214 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-var-lib-kubelet\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-var-lib-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-cnibin\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395ceee6-e47f-4442-8897-6b4a0075f68b-tmp-dir\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691151    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-systemd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-device-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5d0e425-31ca-4ab8-bd99-8534f583d901-serviceca\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-system-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-cnibin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-kubernetes\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-node-log\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691259    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-os-release\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-k8s-cni-cncf-io\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-modprobe-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6681f43-e63d-495d-9870-62d20624cac7-konnectivity-ca\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691309    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.693033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691358    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/395ceee6-e47f-4442-8897-6b4a0075f68b-hosts-file\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-ovn\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-etc-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691419    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-device-dir\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691444    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-netns\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-cnibin\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-etc-selinux\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691321    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-etc-openvswitch\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-var-lib-cni-bin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-netd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-cni-netd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-os-release\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22863a35-4dda-44cb-ae0a-b922feaed80b-ovn-node-metrics-cert\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-cnibin\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzt9\" (UniqueName: \"kubernetes.io/projected/d8317e36-4f62-40a7-91cd-0b523511bbb2-kube-api-access-zfzt9\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-kubernetes\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b5d0e425-31ca-4ab8-bd99-8534f583d901-serviceca\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.693840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395ceee6-e47f-4442-8897-6b4a0075f68b-tmp-dir\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/74670a37-c1b7-4727-9de0-8281ca2176c1-kube-api-access-c2vz2\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-host-run-k8s-cni-cncf-io\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-systemd-units\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-socket-dir-parent\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-run-systemd\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-etc-kubernetes\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-etc-kubernetes\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-dbus\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-node-log\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691826    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-modprobe-d\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-systemd-units\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-lib-modules\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.690119    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691891    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/74670a37-c1b7-4727-9de0-8281ca2176c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691953    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysconfig\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.691961    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:18.694847 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.691983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-socket-dir-parent\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.692025    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:19.192011232 +0000 UTC m=+3.092316433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.692029    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.692093    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:19.192081131 +0000 UTC m=+3.092386332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692098    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-log-socket\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-run-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22863a35-4dda-44cb-ae0a-b922feaed80b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-lib-modules\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a6681f43-e63d-495d-9870-62d20624cac7-konnectivity-ca\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-sysctl-conf\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-conf-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692721    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22863a35-4dda-44cb-ae0a-b922feaed80b-ovnkube-config\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/74670a37-c1b7-4727-9de0-8281ca2176c1-os-release\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e5997655-1f2d-49b4-b556-a1423236c106-iptables-alerter-script\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.692966    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8317e36-4f62-40a7-91cd-0b523511bbb2-multus-cni-dir\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.695810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.693049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8095de59-42e6-4f38-9280-7761a61ef4ae-dbus\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:18.698111 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.697222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-tmp\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.698111 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.697296    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22863a35-4dda-44cb-ae0a-b922feaed80b-ovn-node-metrics-cert\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.698111 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.697582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-etc-tuned\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.700836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.698400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a6681f43-e63d-495d-9870-62d20624cac7-agent-certs\") pod \"konnectivity-agent-r246c\" (UID: \"a6681f43-e63d-495d-9870-62d20624cac7\") " pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.704002 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.703976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw6v\" (UniqueName: \"kubernetes.io/projected/395ceee6-e47f-4442-8897-6b4a0075f68b-kube-api-access-fpw6v\") pod \"node-resolver-hjqwh\" (UID: \"395ceee6-e47f-4442-8897-6b4a0075f68b\") " pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.704107 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.703982    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:18.704107 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.704065    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:18.704107 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.704083    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:18.704387 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:18.704199    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:19.204178084 +0000 UTC m=+3.104483296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:18.707741 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.707614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwvbg\" (UniqueName: \"kubernetes.io/projected/f9c80eb9-4f77-4d1e-aebe-e8c80099c052-kube-api-access-cwvbg\") pod \"aws-ebs-csi-driver-node-mt6kd\" (UID: \"f9c80eb9-4f77-4d1e-aebe-e8c80099c052\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.707741 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.707690    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgrh\" (UniqueName: \"kubernetes.io/projected/22863a35-4dda-44cb-ae0a-b922feaed80b-kube-api-access-wtgrh\") pod \"ovnkube-node-fqfdp\" (UID: \"22863a35-4dda-44cb-ae0a-b922feaed80b\") " pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.708312 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.708286    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vz2\" (UniqueName: \"kubernetes.io/projected/74670a37-c1b7-4727-9de0-8281ca2176c1-kube-api-access-c2vz2\") pod \"multus-additional-cni-plugins-6ww8b\" (UID: \"74670a37-c1b7-4727-9de0-8281ca2176c1\") " pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.708417 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.708396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzt9\" (UniqueName: \"kubernetes.io/projected/d8317e36-4f62-40a7-91cd-0b523511bbb2-kube-api-access-zfzt9\") pod \"multus-4cnx8\" (UID: \"d8317e36-4f62-40a7-91cd-0b523511bbb2\") " pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.708779 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.708760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdkq\" (UniqueName: \"kubernetes.io/projected/d3188e2d-8d2e-41d7-a69a-59e5401c90e0-kube-api-access-4fdkq\") pod \"tuned-zktcq\" (UID: \"d3188e2d-8d2e-41d7-a69a-59e5401c90e0\") " pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.709085 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.708984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8nr\" (UniqueName: \"kubernetes.io/projected/2b72c49e-6395-4d9f-9495-d52457c4680c-kube-api-access-6n8nr\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:18.709316 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.709295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvxw\" (UniqueName: \"kubernetes.io/projected/b5d0e425-31ca-4ab8-bd99-8534f583d901-kube-api-access-fdvxw\") pod \"node-ca-lnq7j\" (UID: \"b5d0e425-31ca-4ab8-bd99-8534f583d901\") " pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.709383 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.709304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsh2j\" (UniqueName: \"kubernetes.io/projected/e5997655-1f2d-49b4-b556-a1423236c106-kube-api-access-nsh2j\") pod \"iptables-alerter-r6cd4\" (UID: \"e5997655-1f2d-49b4-b556-a1423236c106\") " pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.784853 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.784762    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal" event={"ID":"0a73838634c65a1c0797837ab62c2d67","Type":"ContainerStarted","Data":"5e308723e8b716710bb18680d6937ffa724c48197e216ad85fcd28acf45b48f9"}
Jul 10 20:21:18.786346 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.786319    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal" event={"ID":"869786661d939b62b49e72c2e18cdc59","Type":"ContainerStarted","Data":"bcb8a1d9bf53e13bef12e1bde7f7f1eb64bf5c9fa1d56c894c3bccda6cbad2c5"}
Jul 10 20:21:18.850969 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.850943    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:18.857690 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.857579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd"
Jul 10 20:21:18.860700 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.860677    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22863a35_4dda_44cb_ae0a_b922feaed80b.slice/crio-14b45982367174d3a9ed8f75d502359874adeb50136f254b2b4b43338d9d6157 WatchSource:0}: Error finding container 14b45982367174d3a9ed8f75d502359874adeb50136f254b2b4b43338d9d6157: Status 404 returned error can't find the container with id 14b45982367174d3a9ed8f75d502359874adeb50136f254b2b4b43338d9d6157
Jul 10 20:21:18.866970 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.866941    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnq7j"
Jul 10 20:21:18.867578 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.867552    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c80eb9_4f77_4d1e_aebe_e8c80099c052.slice/crio-cd14f1d479ccac5f65b7fadcda5bd88ba3f86ebd2f612f6e3eb6b02e71c2b410 WatchSource:0}: Error finding container cd14f1d479ccac5f65b7fadcda5bd88ba3f86ebd2f612f6e3eb6b02e71c2b410: Status 404 returned error can't find the container with id cd14f1d479ccac5f65b7fadcda5bd88ba3f86ebd2f612f6e3eb6b02e71c2b410
Jul 10 20:21:18.872726 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.872702    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:18.872812 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.872796    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zktcq"
Jul 10 20:21:18.877186 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.877009    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 10 20:21:18.878345 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.878319    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d0e425_31ca_4ab8_bd99_8534f583d901.slice/crio-21e0c1d6fe6a948bf415d6f61a9fd933931a7b4082dc9f759670ba5b809453af WatchSource:0}: Error finding container 21e0c1d6fe6a948bf415d6f61a9fd933931a7b4082dc9f759670ba5b809453af: Status 404 returned error can't find the container with id 21e0c1d6fe6a948bf415d6f61a9fd933931a7b4082dc9f759670ba5b809453af
Jul 10 20:21:18.879530 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.879508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6ww8b"
Jul 10 20:21:18.884068 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.884039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3188e2d_8d2e_41d7_a69a_59e5401c90e0.slice/crio-679fe868ecfe3af69684fb89110152247dbdd963ba4941b3a273e0965fdb74c3 WatchSource:0}: Error finding container 679fe868ecfe3af69684fb89110152247dbdd963ba4941b3a273e0965fdb74c3: Status 404 returned error can't find the container with id 679fe868ecfe3af69684fb89110152247dbdd963ba4941b3a273e0965fdb74c3
Jul 10 20:21:18.886050 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.886032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4cnx8"
Jul 10 20:21:18.892852 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.892821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hjqwh"
Jul 10 20:21:18.899234 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.899217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:18.904777 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:18.904749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-r6cd4"
Jul 10 20:21:18.905234 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.905209    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395ceee6_e47f_4442_8897_6b4a0075f68b.slice/crio-6c7d720c2ead7ffda03f486c76c503dccb59799334c25a1ba5f21a27565ab41f WatchSource:0}: Error finding container 6c7d720c2ead7ffda03f486c76c503dccb59799334c25a1ba5f21a27565ab41f: Status 404 returned error can't find the container with id 6c7d720c2ead7ffda03f486c76c503dccb59799334c25a1ba5f21a27565ab41f
Jul 10 20:21:18.910732 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.910709    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6681f43_e63d_495d_9870_62d20624cac7.slice/crio-5d8e4968db43750dd5a7034830fb0f7af11b00486ffc5af51cf38c09a2072554 WatchSource:0}: Error finding container 5d8e4968db43750dd5a7034830fb0f7af11b00486ffc5af51cf38c09a2072554: Status 404 returned error can't find the container with id 5d8e4968db43750dd5a7034830fb0f7af11b00486ffc5af51cf38c09a2072554
Jul 10 20:21:18.917082 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:18.917063    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5997655_1f2d_49b4_b556_a1423236c106.slice/crio-f2ebd33597e93b968841538d39c01ec572ddf22f8fe4ce1d82bc7b61b20a7d2a WatchSource:0}: Error finding container f2ebd33597e93b968841538d39c01ec572ddf22f8fe4ce1d82bc7b61b20a7d2a: Status 404 returned error can't find the container with id f2ebd33597e93b968841538d39c01ec572ddf22f8fe4ce1d82bc7b61b20a7d2a
Jul 10 20:21:19.195558 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.195522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:19.195740 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.195569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:19.195740 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.195692    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:19.195851 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.195737    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:19.195851 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.195774    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:20.195751067 +0000 UTC m=+4.096056277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:19.195851 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.195793    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:20.195784529 +0000 UTC m=+4.096089737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:19.296718 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.296687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:19.296899 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.296871    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:19.296899 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.296893    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:19.296899 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.296903    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:19.297037 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.296960    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:20.296945386 +0000 UTC m=+4.197250583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:19.614678 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.614633    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:16:18 +0000 UTC" deadline="2028-01-24 18:48:50.835676965 +0000 UTC"
Jul 10 20:21:19.614678 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.614671    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13510h27m31.221011073s"
Jul 10 20:21:19.778719 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.778683    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:19.779293 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.778805    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:19.779293 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.778904    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:19.779293 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:19.779025    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:19.791761 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.791729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6cd4" event={"ID":"e5997655-1f2d-49b4-b556-a1423236c106","Type":"ContainerStarted","Data":"f2ebd33597e93b968841538d39c01ec572ddf22f8fe4ce1d82bc7b61b20a7d2a"}
Jul 10 20:21:19.793209 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.793183    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4cnx8" event={"ID":"d8317e36-4f62-40a7-91cd-0b523511bbb2","Type":"ContainerStarted","Data":"7d42c2dece5facf1c7dff0248f019bd52903b252a30ccbfe096e27d4f519c846"}
Jul 10 20:21:19.794392 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.794369    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd" event={"ID":"f9c80eb9-4f77-4d1e-aebe-e8c80099c052","Type":"ContainerStarted","Data":"cd14f1d479ccac5f65b7fadcda5bd88ba3f86ebd2f612f6e3eb6b02e71c2b410"}
Jul 10 20:21:19.795535 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.795508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"14b45982367174d3a9ed8f75d502359874adeb50136f254b2b4b43338d9d6157"}
Jul 10 20:21:19.796856 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.796802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r246c" event={"ID":"a6681f43-e63d-495d-9870-62d20624cac7","Type":"ContainerStarted","Data":"5d8e4968db43750dd5a7034830fb0f7af11b00486ffc5af51cf38c09a2072554"}
Jul 10 20:21:19.798407 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.798353    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjqwh" event={"ID":"395ceee6-e47f-4442-8897-6b4a0075f68b","Type":"ContainerStarted","Data":"6c7d720c2ead7ffda03f486c76c503dccb59799334c25a1ba5f21a27565ab41f"}
Jul 10 20:21:19.799853 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.799830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerStarted","Data":"c18099cea36607067942d22723078bebc1123cfc4e7f3de63033795e5e1a5708"}
Jul 10 20:21:19.801412 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.801357    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zktcq" event={"ID":"d3188e2d-8d2e-41d7-a69a-59e5401c90e0","Type":"ContainerStarted","Data":"679fe868ecfe3af69684fb89110152247dbdd963ba4941b3a273e0965fdb74c3"}
Jul 10 20:21:19.802719 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:19.802674    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnq7j" event={"ID":"b5d0e425-31ca-4ab8-bd99-8534f583d901","Type":"ContainerStarted","Data":"21e0c1d6fe6a948bf415d6f61a9fd933931a7b4082dc9f759670ba5b809453af"}
Jul 10 20:21:20.204461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.204233    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:20.204461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.204292    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:20.204461 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.204393    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:20.204461 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.204415    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:20.204797 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.204479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:22.204458483 +0000 UTC m=+6.104763687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:20.204797 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.204516    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:22.204505673 +0000 UTC m=+6.104810883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:20.305101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.305062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:20.305277 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.305235    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:20.305277 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.305257    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:20.305277 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.305269    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:20.305446 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.305330    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:22.305309332 +0000 UTC m=+6.205614542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:20.615393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.615352    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 20:16:18 +0000 UTC" deadline="2028-03-05 21:45:46.693944009 +0000 UTC"
Jul 10 20:21:20.615393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.615392    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14497h24m26.078556897s"
Jul 10 20:21:20.781646 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:20.781613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:20.782079 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:20.781745    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:21.778553 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:21.778518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:21.778735 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:21.778664    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:21.779118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:21.779095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:21.779225 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:21.779201    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:22.221642 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:22.221607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:22.222078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:22.221663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:22.222078 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.221740    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:22.222078 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.221765    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:22.222078 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.221812    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:26.221790309 +0000 UTC m=+10.122095507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:22.222078 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.221833    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:26.221822546 +0000 UTC m=+10.122127747 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:22.322606 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:22.322572    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:22.322784 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.322726    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:22.322784 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.322743    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:22.322784 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.322756    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:22.322939 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.322810    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:26.322792697 +0000 UTC m=+10.223097900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:22.782635 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:22.781621    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:22.782635 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:22.781747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:23.779273 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:23.779240    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:23.779773 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:23.779374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:23.779773 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:23.779440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:23.779773 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:23.779536    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:24.781729 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:24.781633    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:24.782092 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:24.781754    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:25.778544 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:25.778382    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:25.778544 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:25.778416    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:25.778544 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:25.778537    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:25.778942 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:25.778667    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:26.254481 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:26.254417    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:26.254481 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:26.254481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:26.255081 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.254631    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:26.255081 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.254697    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:34.25467716 +0000 UTC m=+18.154982377 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:26.255192 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.255108    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:26.255192 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.255162    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:34.255145746 +0000 UTC m=+18.155450959 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:26.355843 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:26.355729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:26.356004 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.355918    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:26.356004 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.355947    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:26.356004 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.355962    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:26.356234 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.356021    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:34.356001162 +0000 UTC m=+18.256306382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:26.780122 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:26.780089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:26.780300 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:26.780167    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:27.779675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:27.779028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:27.779675 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:27.779157    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:27.779675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:27.779032    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:27.779675 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:27.779582    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:28.782025 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:28.782000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:28.782473 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:28.782095    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:29.778850 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:29.778819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:29.779015 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:29.778936    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:29.779015 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:29.778989    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:29.779138 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:29.779101    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:30.781107 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:30.781081    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:30.781515 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:30.781181    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:31.779100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:31.779063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:31.779100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:31.779105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:31.779310 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:31.779198    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:31.779368 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:31.779339    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:32.778621 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:32.778589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:32.779014 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:32.778698    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:33.779253 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:33.779217    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:33.779796 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:33.779313    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:33.779796 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:33.779342    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:33.779796 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:33.779454    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:34.315012 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:34.314978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:34.315195 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:34.315064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:34.315195 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.315181    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:34.315296 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.315236    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:21:50.315216646 +0000 UTC m=+34.215521846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:34.315501 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.315467    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:34.315599 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.315554    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:21:50.315536477 +0000 UTC m=+34.215841681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:34.415561 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:34.415514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:34.415709 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.415689    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:34.415787 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.415716    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:34.415787 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.415730    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:34.415894 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.415793    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:50.415773848 +0000 UTC m=+34.316079060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:34.779372 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:34.779284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:34.779825 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:34.779419    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:35.779081 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:35.779049    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:35.779267 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:35.779167    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:35.779267 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:35.779219    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:35.779364 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:35.779320    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:36.779606 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:36.779575    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:36.780011 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:36.779685    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:37.779248 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:37.779224    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:37.779332 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:37.779319    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:37.779394 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:37.779376    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:37.779483 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:37.779468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:37.835983 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:37.835952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal" event={"ID":"869786661d939b62b49e72c2e18cdc59","Type":"ContainerStarted","Data":"6015a462df4da0d6b7fa032fee357421ca2a326e8423c309853a77477b823388"}
Jul 10 20:21:38.779237 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.778844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:38.779384 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:38.779333    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:38.839852 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.839823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zktcq" event={"ID":"d3188e2d-8d2e-41d7-a69a-59e5401c90e0","Type":"ContainerStarted","Data":"1e066988a4839f7338743764ad53deb14d7a4c472226f9d90d77b5c19ea29269"}
Jul 10 20:21:38.841504 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.841472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnq7j" event={"ID":"b5d0e425-31ca-4ab8-bd99-8534f583d901","Type":"ContainerStarted","Data":"058f5d6bf287bd5ed623528bee9ec4d02a153886e8028a4fe8b94004373038e5"}
Jul 10 20:21:38.843179 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.843148    2564 generic.go:356] "Generic (PLEG): container finished" podID="0a73838634c65a1c0797837ab62c2d67" containerID="d38441b956082ef4344be53c44f0d5e203c85a6b0ae9c107c6f6f7b9b3739de0" exitCode=0
Jul 10 20:21:38.843290 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.843229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal" event={"ID":"0a73838634c65a1c0797837ab62c2d67","Type":"ContainerDied","Data":"d38441b956082ef4344be53c44f0d5e203c85a6b0ae9c107c6f6f7b9b3739de0"}
Jul 10 20:21:38.845127 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.845102    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4cnx8" event={"ID":"d8317e36-4f62-40a7-91cd-0b523511bbb2","Type":"ContainerStarted","Data":"4e470a57de70c8377e79b3449599bad17fd76c9c5cdf60aea128cdc6f0ba7c65"}
Jul 10 20:21:38.846848 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.846823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd" event={"ID":"f9c80eb9-4f77-4d1e-aebe-e8c80099c052","Type":"ContainerStarted","Data":"08c26be7b89676af9273041c3be33577333cbdd917530c25e14013036692321d"}
Jul 10 20:21:38.849837 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.849814    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:21:38.850202 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850175    2564 generic.go:356] "Generic (PLEG): container finished" podID="22863a35-4dda-44cb-ae0a-b922feaed80b" containerID="de02b031761678623c883b867ff0724d88db2d108194cd998e0e1c53188fa0b0" exitCode=1
Jul 10 20:21:38.850298 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"688796ba62fd1903ecdc58bbf91d1f9fd926a06c40a2e1a90f89bdab1a6bdaf1"}
Jul 10 20:21:38.850363 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"268e1c74a28c1aad41c2447b98505a20332ca3eb2352a22cb9b995759c7a25b3"}
Jul 10 20:21:38.850427 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"a204a14c3168a8eef159423498e5ac0bf02ecf250dfbd95470237792d4321bdc"}
Jul 10 20:21:38.850484 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"a4e0f34166a2c2cc9f29f74763442949e6ea235e12377c3ee732b90bc1a3f730"}
Jul 10 20:21:38.850484 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerDied","Data":"de02b031761678623c883b867ff0724d88db2d108194cd998e0e1c53188fa0b0"}
Jul 10 20:21:38.850609 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.850544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"fa3b8194297a68286097c8b57ead214cb8e43c3dfb5742e549652e452718901d"}
Jul 10 20:21:38.851736 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.851717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-r246c" event={"ID":"a6681f43-e63d-495d-9870-62d20624cac7","Type":"ContainerStarted","Data":"a5aa4c96c2d1256acfe1b402888c3d650b57ca6ef2fc6af394322cc63d5750ee"}
Jul 10 20:21:38.853094 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.853072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hjqwh" event={"ID":"395ceee6-e47f-4442-8897-6b4a0075f68b","Type":"ContainerStarted","Data":"fe1efb60d40eda3c323580cd5a710eb50a579c55374283134bf30921d49b91b1"}
Jul 10 20:21:38.854687 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.854653    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="1b5aaaeb867bb2f9e9900d554bbcf5511a547c017dfc6ac0351853eb9c21f6fc" exitCode=0
Jul 10 20:21:38.854687 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.854678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"1b5aaaeb867bb2f9e9900d554bbcf5511a547c017dfc6ac0351853eb9c21f6fc"}
Jul 10 20:21:38.857165 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.857126    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zktcq" podStartSLOduration=4.063723426 podStartE2EDuration="22.857113014s" totalImagesPullingTime="18.793389588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:38.856437013 +0000 UTC m=+22.756742233" watchObservedRunningTime="2026-07-10 20:21:38.857113014 +0000 UTC m=+22.757418233"
Jul 10 20:21:38.869861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.869823    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-r246c" podStartSLOduration=3.3484995 podStartE2EDuration="21.869809339s" totalImagesPullingTime="18.521309839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:38.869668088 +0000 UTC m=+22.769973309" watchObservedRunningTime="2026-07-10 20:21:38.869809339 +0000 UTC m=+22.770114560"
Jul 10 20:21:38.949509 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.949444    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lnq7j" podStartSLOduration=8.588938831 podStartE2EDuration="22.949425123s" totalImagesPullingTime="14.360486292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:38.913910068 +0000 UTC m=+22.814215288" watchObservedRunningTime="2026-07-10 20:21:38.949425123 +0000 UTC m=+22.849730343"
Jul 10 20:21:38.963754 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:38.963717    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-240.ec2.internal" podStartSLOduration=21.963704084 podStartE2EDuration="21.963704084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:38.963423293 +0000 UTC m=+22.863728538" watchObservedRunningTime="2026-07-10 20:21:38.963704084 +0000 UTC m=+22.864009303"
Jul 10 20:21:39.002814 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.002762    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4cnx8" podStartSLOduration=2.7090328059999997 podStartE2EDuration="22.002747856s" totalImagesPullingTime="19.29371505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:39.002332107 +0000 UTC m=+22.902637325" watchObservedRunningTime="2026-07-10 20:21:39.002747856 +0000 UTC m=+22.903053075"
Jul 10 20:21:39.017441 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.017390    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hjqwh" podStartSLOduration=3.242825275 podStartE2EDuration="22.017370943s" totalImagesPullingTime="18.774545668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:39.016763882 +0000 UTC m=+22.917069102" watchObservedRunningTime="2026-07-10 20:21:39.017370943 +0000 UTC m=+22.917676162"
Jul 10 20:21:39.601877 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.601839    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 10 20:21:39.677081 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.676891    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T20:21:39.601859395Z","UUID":"d5c24495-ff77-4659-aacb-d9bc12641898","Handler":null,"Name":"","Endpoint":""}
Jul 10 20:21:39.678608 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.678572    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 10 20:21:39.678762 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.678618    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 10 20:21:39.779227 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.779153    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:39.779397 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.779165    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:39.779397 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:39.779281    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:39.779514 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:39.779398    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:39.859123 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.859055    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal" event={"ID":"0a73838634c65a1c0797837ab62c2d67","Type":"ContainerStarted","Data":"7ba5f90c357f3c89c34ca5d0be0c3bc56c6762528f7b305195a53f0cff9298ff"}
Jul 10 20:21:39.860797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.860772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-r6cd4" event={"ID":"e5997655-1f2d-49b4-b556-a1423236c106","Type":"ContainerStarted","Data":"5911cca61084d9337956bd04d690550c4cbfb7fff03cfced77091694f256a68a"}
Jul 10 20:21:39.862573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.862549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd" event={"ID":"f9c80eb9-4f77-4d1e-aebe-e8c80099c052","Type":"ContainerStarted","Data":"ad7aadd06cd97c74fa04e6a1f442e59d716705de8bd78f2b6cf6219942133ccd"}
Jul 10 20:21:39.874309 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.874261    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-240.ec2.internal" podStartSLOduration=22.874246941 podStartE2EDuration="22.874246941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:39.87386972 +0000 UTC m=+23.774174939" watchObservedRunningTime="2026-07-10 20:21:39.874246941 +0000 UTC m=+23.774552160"
Jul 10 20:21:39.888473 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:39.888421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-r6cd4" podStartSLOduration=4.3727485680000004 podStartE2EDuration="22.888400313s" totalImagesPullingTime="18.515651745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:39.887819873 +0000 UTC m=+23.788125104" watchObservedRunningTime="2026-07-10 20:21:39.888400313 +0000 UTC m=+23.788705534"
Jul 10 20:21:40.779419 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:40.779386    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:40.779649 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:40.779541    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:40.784649 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:40.784626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:40.785251 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:40.785226    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:40.866238 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:40.866220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:40.866698 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:40.866682    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-r246c"
Jul 10 20:21:41.779181 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.779001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:41.779381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.779023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:41.779381 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:41.779282    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:41.779484 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:41.779396    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:41.870295 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.870250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd" event={"ID":"f9c80eb9-4f77-4d1e-aebe-e8c80099c052","Type":"ContainerStarted","Data":"e5492694f23e4c9f46676ef46faead667d736df7b5d68fab9c6699018e5501a3"}
Jul 10 20:21:41.873506 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.873474    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:21:41.873933 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.873911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"450214e3ddf4ac88277d7c78217ee5c058b4b6defa6547eb23443ca6d2acdd54"}
Jul 10 20:21:41.891640 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:41.891579    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mt6kd" podStartSLOduration=4.967798091 podStartE2EDuration="25.89156079s" totalImagesPullingTime="20.923762699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:41.891125997 +0000 UTC m=+25.791431217" watchObservedRunningTime="2026-07-10 20:21:41.89156079 +0000 UTC m=+25.791866016"
Jul 10 20:21:42.779196 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:42.779160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:42.779376 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:42.779301    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:43.779226 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.779003    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:43.779897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.779002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:43.779897 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:43.779260    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:43.779897 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:43.779338    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:43.879936 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.879906    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="6c70aa20de20ddcd045c28e2ff26bffad7eb22ed547a87f92394c2e8ad9a56f6" exitCode=0
Jul 10 20:21:43.880095 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.879992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"6c70aa20de20ddcd045c28e2ff26bffad7eb22ed547a87f92394c2e8ad9a56f6"}
Jul 10 20:21:43.883595 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.883579    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:21:43.883937 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.883916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"e3c69b559490eeafe560c07c06f58a7a28e1f30ee01a3e294621dc4b20045523"}
Jul 10 20:21:43.884175 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.884161    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:43.884365 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.884348    2564 scope.go:117] "RemoveContainer" containerID="de02b031761678623c883b867ff0724d88db2d108194cd998e0e1c53188fa0b0"
Jul 10 20:21:43.902891 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:43.902857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:44.781927 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.781900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:44.782329 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:44.781993    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:44.889430 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.889406    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:21:44.889792 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.889766    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" event={"ID":"22863a35-4dda-44cb-ae0a-b922feaed80b","Type":"ContainerStarted","Data":"eb4f63c8f5ade61fe9f11297b3aef86beec88342282ee8abaa4332121073323f"}
Jul 10 20:21:44.889980 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.889958    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:44.890091 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.890041    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:44.905397 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.905372    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:21:44.918118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:44.918068    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp" podStartSLOduration=10.057616623 podStartE2EDuration="28.918052891s" totalImagesPullingTime="18.860436268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:44.917760662 +0000 UTC m=+28.818065895" watchObservedRunningTime="2026-07-10 20:21:44.918052891 +0000 UTC m=+28.818358109"
Jul 10 20:21:45.199552 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.199529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqc7d"]
Jul 10 20:21:45.199667 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.199653    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:45.199758 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:45.199738    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:45.203141 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.203034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfp74"]
Jul 10 20:21:45.203312 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.203292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:45.203440 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:45.203415    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:45.203676 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.203660    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ml7ds"]
Jul 10 20:21:45.203772 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.203745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:45.203851 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:45.203829    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:45.893775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.893738    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="d507f83bab82f9690636d147d29c22860188ac26458ccd0b11fa5de2734046b2" exitCode=0
Jul 10 20:21:45.894144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:45.893828    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"d507f83bab82f9690636d147d29c22860188ac26458ccd0b11fa5de2734046b2"}
Jul 10 20:21:46.779316 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:46.779283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:46.779537 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:46.779368    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:46.779537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:46.779469    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:46.779623 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:46.779586    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:46.779623 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:46.779613    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:46.779683 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:46.779667    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:47.901185 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:47.901020    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="487774b1d042db5cc9067ebf5c4799ec3ad17554d54d3852ce4d21c92abc8b5f" exitCode=0
Jul 10 20:21:47.901600 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:47.901099    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"487774b1d042db5cc9067ebf5c4799ec3ad17554d54d3852ce4d21c92abc8b5f"}
Jul 10 20:21:48.778929 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:48.778890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:48.779112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:48.778890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:48.779112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:48.779039    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:48.779112 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:48.779037    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:48.779273 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:48.779112    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:48.779273 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:48.779190    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:50.335021 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.334990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:50.335536 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.335036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:50.335536 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.335132    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:50.335536 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.335147    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:50.335536 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.335191    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:22:22.335177592 +0000 UTC m=+66.235482788 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 10 20:21:50.335536 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.335206    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret podName:8095de59-42e6-4f38-9280-7761a61ef4ae nodeName:}" failed. No retries permitted until 2026-07-10 20:22:22.335200036 +0000 UTC m=+66.235505232 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret") pod "global-pull-secret-syncer-gqc7d" (UID: "8095de59-42e6-4f38-9280-7761a61ef4ae") : object "kube-system"/"original-pull-secret" not registered
Jul 10 20:21:50.436430 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.436388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:50.436608 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.436582    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 10 20:21:50.436608 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.436604    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 10 20:21:50.436717 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.436616    2564 projected.go:196] Error preparing data for projected volume kube-api-access-ktl2m for pod openshift-network-diagnostics/network-check-target-ml7ds: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:50.436717 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.436682    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m podName:8768c4f5-0443-4905-9d52-3b7b4ac32e09 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:22.436663131 +0000 UTC m=+66.336968347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktl2m" (UniqueName: "kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m") pod "network-check-target-ml7ds" (UID: "8768c4f5-0443-4905-9d52-3b7b4ac32e09") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 10 20:21:50.778959 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.778923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:50.779125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.778924    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:50.779125 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.779054    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqc7d" podUID="8095de59-42e6-4f38-9280-7761a61ef4ae"
Jul 10 20:21:50.779125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.779073    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:50.779280 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.779174    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ml7ds" podUID="8768c4f5-0443-4905-9d52-3b7b4ac32e09"
Jul 10 20:21:50.779335 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:50.779267    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:21:50.967647 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.967556    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-240.ec2.internal" event="NodeReady"
Jul 10 20:21:50.967788 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:50.967705    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 10 20:21:51.000309 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.000278    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:21:51.005317 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.005295    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"]
Jul 10 20:21:51.005438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.005417    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.008336 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008312    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 10 20:21:51.008635 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008618    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 10 20:21:51.008740 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008641    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s5ppj\""
Jul 10 20:21:51.008740 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008641    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"]
Jul 10 20:21:51.008740 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008738    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 10 20:21:51.008892 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.008739    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.011385 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.011366    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 10 20:21:51.011730 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.011706    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 10 20:21:51.011859 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.011739    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 10 20:21:51.011926 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.011861    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 10 20:21:51.012113 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.012095    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"]
Jul 10 20:21:51.012550 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.012531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.014658 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.014636    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 10 20:21:51.015362 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.015342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.015770 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.015648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 10 20:21:51.015770 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.015676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vwx2c\""
Jul 10 20:21:51.016256 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.016227    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"]
Jul 10 20:21:51.018083 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.018061    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 10 20:21:51.018762 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.018744    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 10 20:21:51.019029 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.019012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 10 20:21:51.019865 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.019645    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 10 20:21:51.019865 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.019827    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"]
Jul 10 20:21:51.020021 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.019998    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:21:51.024578 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.023799    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nc87d"]
Jul 10 20:21:51.027125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.027108    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r2j9g"]
Jul 10 20:21:51.027224 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.027207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.029957 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.029804    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 10 20:21:51.029957 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.029882    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgw85\""
Jul 10 20:21:51.029957 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.029923    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 10 20:21:51.030531 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.030511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.033048 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.033023    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xpb5f\""
Jul 10 20:21:51.033138 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.033104    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 10 20:21:51.036059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.034623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 10 20:21:51.036059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.034827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 10 20:21:51.036059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.035514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"]
Jul 10 20:21:51.040246 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.040179    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2j9g"]
Jul 10 20:21:51.041560 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.041525    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nc87d"]
Jul 10 20:21:51.142235 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142203    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.142235 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142241    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-cert\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142311    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j68jt\" (UniqueName: \"kubernetes.io/projected/190eea48-564e-4b13-bc2e-9dec8aba9dd7-kube-api-access-j68jt\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142360    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.142469 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6df\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4qj\" (UniqueName: \"kubernetes.io/projected/4af0945f-e5ce-416e-90dd-2441b84f8afd-kube-api-access-9x4qj\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58873ee7-32c0-4d3b-a18a-3ab6fd149297-config-volume\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5v82\" (UniqueName: \"kubernetes.io/projected/856a7f25-761a-4874-bf5f-1be0594090e2-kube-api-access-k5v82\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58873ee7-32c0-4d3b-a18a-3ab6fd149297-tmp-dir\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142662    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj424\" (UniqueName: \"kubernetes.io/projected/58873ee7-32c0-4d3b-a18a-3ab6fd149297-kube-api-access-mj424\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856a7f25-761a-4874-bf5f-1be0594090e2-tmp\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.142746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/190eea48-564e-4b13-bc2e-9dec8aba9dd7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af0945f-e5ce-416e-90dd-2441b84f8afd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142927    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45bc8\" (UniqueName: \"kubernetes.io/projected/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-kube-api-access-45bc8\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.143101 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.142959    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/856a7f25-761a-4874-bf5f-1be0594090e2-klusterlet-config\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.243629 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.243629 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj424\" (UniqueName: \"kubernetes.io/projected/58873ee7-32c0-4d3b-a18a-3ab6fd149297-kube-api-access-mj424\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.243838 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856a7f25-761a-4874-bf5f-1be0594090e2-tmp\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.243838 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.243941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/190eea48-564e-4b13-bc2e-9dec8aba9dd7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.243941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.243941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243918    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.244076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.243957    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af0945f-e5ce-416e-90dd-2441b84f8afd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.244318 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/856a7f25-761a-4874-bf5f-1be0594090e2-tmp\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.244516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45bc8\" (UniqueName: \"kubernetes.io/projected/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-kube-api-access-45bc8\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.244516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/856a7f25-761a-4874-bf5f-1be0594090e2-klusterlet-config\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.244516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244481    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-cert\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244559    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j68jt\" (UniqueName: \"kubernetes.io/projected/190eea48-564e-4b13-bc2e-9dec8aba9dd7-kube-api-access-j68jt\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244627    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/190eea48-564e-4b13-bc2e-9dec8aba9dd7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244645    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.244694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.244973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244701    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.244973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244716    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.245061 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.245029    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:21:51.245061 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.245045    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:21:51.245143 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.245143 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.245110    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:51.745090741 +0000 UTC m=+35.645395952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:21:51.245411 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.244725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.245411 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.245411 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245369    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.245411 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245397    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6df\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.245648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245424    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4qj\" (UniqueName: \"kubernetes.io/projected/4af0945f-e5ce-416e-90dd-2441b84f8afd-kube-api-access-9x4qj\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.245648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245448    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58873ee7-32c0-4d3b-a18a-3ab6fd149297-config-volume\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.245648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5v82\" (UniqueName: \"kubernetes.io/projected/856a7f25-761a-4874-bf5f-1be0594090e2-kube-api-access-k5v82\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.245648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58873ee7-32c0-4d3b-a18a-3ab6fd149297-tmp-dir\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.245648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.245574    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.247046 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.246187    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:21:51.247046 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.246250    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:51.746234558 +0000 UTC m=+35.646539757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:21:51.247046 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.246924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58873ee7-32c0-4d3b-a18a-3ab6fd149297-tmp-dir\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.247046 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.246996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58873ee7-32c0-4d3b-a18a-3ab6fd149297-config-volume\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.249527 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/856a7f25-761a-4874-bf5f-1be0594090e2-klusterlet-config\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.249527 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4af0945f-e5ce-416e-90dd-2441b84f8afd-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.249700 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249636    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.249700 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-ca\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.249810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.249810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249719    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.249920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249846    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.249971 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.249945    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/190eea48-564e-4b13-bc2e-9dec8aba9dd7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.251213 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.251192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-cert\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.257342 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.257323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.258663 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.258642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5v82\" (UniqueName: \"kubernetes.io/projected/856a7f25-761a-4874-bf5f-1be0594090e2-kube-api-access-k5v82\") pod \"klusterlet-addon-workmgr-578d876cdc-4px4r\" (UID: \"856a7f25-761a-4874-bf5f-1be0594090e2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.258920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.258896    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4qj\" (UniqueName: \"kubernetes.io/projected/4af0945f-e5ce-416e-90dd-2441b84f8afd-kube-api-access-9x4qj\") pod \"managed-serviceaccount-addon-agent-6cbff74488-cx4mm\" (UID: \"4af0945f-e5ce-416e-90dd-2441b84f8afd\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.259659 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.259634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj424\" (UniqueName: \"kubernetes.io/projected/58873ee7-32c0-4d3b-a18a-3ab6fd149297-kube-api-access-mj424\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.260381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.259976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45bc8\" (UniqueName: \"kubernetes.io/projected/aa14fe47-a4ae-4b17-8b81-3965e3c2674b-kube-api-access-45bc8\") pod \"ingress-canary-r2j9g\" (UID: \"aa14fe47-a4ae-4b17-8b81-3965e3c2674b\") " pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.260381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.260337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j68jt\" (UniqueName: \"kubernetes.io/projected/190eea48-564e-4b13-bc2e-9dec8aba9dd7-kube-api-access-j68jt\") pod \"cluster-proxy-proxy-agent-54bb96946d-zfbtd\" (UID: \"190eea48-564e-4b13-bc2e-9dec8aba9dd7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.260908 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.260880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6df\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.329420 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.329391    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:21:51.344664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.344639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:21:51.351604 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.351584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:21:51.377612 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.377584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r2j9g"
Jul 10 20:21:51.749431 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.749396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:51.749620 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:51.749449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:51.749620 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.749592    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:21:51.749706 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.749666    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:52.749643132 +0000 UTC m=+36.649948333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:21:51.749706 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.749602    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:21:51.749706 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.749688    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:21:51.749815 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:51.749734    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:52.749721362 +0000 UTC m=+36.650026558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:21:52.098174 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.098030    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"]
Jul 10 20:21:52.101475 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.101448    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r2j9g"]
Jul 10 20:21:52.101936 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:52.101905    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4af0945f_e5ce_416e_90dd_2441b84f8afd.slice/crio-0138a71034f9db95dcc5b84a6da7a5c827fe8fcb5218a04b9775baa3f7cde743 WatchSource:0}: Error finding container 0138a71034f9db95dcc5b84a6da7a5c827fe8fcb5218a04b9775baa3f7cde743: Status 404 returned error can't find the container with id 0138a71034f9db95dcc5b84a6da7a5c827fe8fcb5218a04b9775baa3f7cde743
Jul 10 20:21:52.105971 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:52.105949    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa14fe47_a4ae_4b17_8b81_3965e3c2674b.slice/crio-585d9c65bbcdd2cf1e02a780e44e7c2ff59a8623c26f2b2e529e8f4ee952a765 WatchSource:0}: Error finding container 585d9c65bbcdd2cf1e02a780e44e7c2ff59a8623c26f2b2e529e8f4ee952a765: Status 404 returned error can't find the container with id 585d9c65bbcdd2cf1e02a780e44e7c2ff59a8623c26f2b2e529e8f4ee952a765
Jul 10 20:21:52.108218 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.108172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"]
Jul 10 20:21:52.110318 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:52.110273    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod190eea48_564e_4b13_bc2e_9dec8aba9dd7.slice/crio-f29f7b6a13d8cef901d8f101210b96cdeba54666d2d8eb4d102d5fc41577693c WatchSource:0}: Error finding container f29f7b6a13d8cef901d8f101210b96cdeba54666d2d8eb4d102d5fc41577693c: Status 404 returned error can't find the container with id f29f7b6a13d8cef901d8f101210b96cdeba54666d2d8eb4d102d5fc41577693c
Jul 10 20:21:52.114290 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.114262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"]
Jul 10 20:21:52.116600 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:21:52.116577    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856a7f25_761a_4874_bf5f_1be0594090e2.slice/crio-3a9f739986072530e448c29fb7f2c8b2c8a4409976a2f86ebcb470def0a82033 WatchSource:0}: Error finding container 3a9f739986072530e448c29fb7f2c8b2c8a4409976a2f86ebcb470def0a82033: Status 404 returned error can't find the container with id 3a9f739986072530e448c29fb7f2c8b2c8a4409976a2f86ebcb470def0a82033
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.757183    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.757252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:52.757398    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:52.757414    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:52.757472    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:54.757452834 +0000 UTC m=+38.657758047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:52.757601    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:21:52.757664 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:52.757638    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:54.757626125 +0000 UTC m=+38.657931324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:21:52.781067 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.779589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:21:52.781067 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.780361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:21:52.782129 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.781503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:21:52.782736 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.782475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 20:21:52.783244 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.783026    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 20:21:52.783756 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.783738    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pk8lx\""
Jul 10 20:21:52.784013 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.783990    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 20:21:52.785243 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.784305    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 20:21:52.785243 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.785070    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr56z\""
Jul 10 20:21:52.918314 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.918166    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerStarted","Data":"3a9f739986072530e448c29fb7f2c8b2c8a4409976a2f86ebcb470def0a82033"}
Jul 10 20:21:52.919818 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.919754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerStarted","Data":"0138a71034f9db95dcc5b84a6da7a5c827fe8fcb5218a04b9775baa3f7cde743"}
Jul 10 20:21:52.921186 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.921129    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerStarted","Data":"f29f7b6a13d8cef901d8f101210b96cdeba54666d2d8eb4d102d5fc41577693c"}
Jul 10 20:21:52.927037 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.927011    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="8c50286a5214a2e2cf3dc825195f688d9593bdc0c5138602a93f9d1e7bd9427c" exitCode=0
Jul 10 20:21:52.927147 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.927086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"8c50286a5214a2e2cf3dc825195f688d9593bdc0c5138602a93f9d1e7bd9427c"}
Jul 10 20:21:52.929649 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:52.929624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2j9g" event={"ID":"aa14fe47-a4ae-4b17-8b81-3965e3c2674b","Type":"ContainerStarted","Data":"585d9c65bbcdd2cf1e02a780e44e7c2ff59a8623c26f2b2e529e8f4ee952a765"}
Jul 10 20:21:53.941394 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:53.940536    2564 generic.go:356] "Generic (PLEG): container finished" podID="74670a37-c1b7-4727-9de0-8281ca2176c1" containerID="8862641d7becff9890da69ebd08f01081050fb93f61a2d0f350218b558ca77da" exitCode=0
Jul 10 20:21:53.941394 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:53.940602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerDied","Data":"8862641d7becff9890da69ebd08f01081050fb93f61a2d0f350218b558ca77da"}
Jul 10 20:21:54.775394 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:54.775143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:54.775567 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:54.775326    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:21:54.775567 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:54.775517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:54.775567 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:54.775529    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:21:54.775723 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:54.775593    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:58.775571207 +0000 UTC m=+42.675876422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:21:54.775723 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:54.775644    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:21:54.775723 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:54.775705    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:21:58.775671092 +0000 UTC m=+42.675976293 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:21:54.950878 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:54.950839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" event={"ID":"74670a37-c1b7-4727-9de0-8281ca2176c1","Type":"ContainerStarted","Data":"823554a275d644b3388f5a50d9d9f15d062a9ad24662b36cb4fb358a00e7eb05"}
Jul 10 20:21:54.978199 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:54.977934    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6ww8b" podStartSLOduration=9.077355501 podStartE2EDuration="38.977911044s" totalImagesPullingTime="29.900555543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:21:54.975458553 +0000 UTC m=+38.875763796" watchObservedRunningTime="2026-07-10 20:21:54.977911044 +0000 UTC m=+38.878216267"
Jul 10 20:21:58.812771 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:58.812713    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:21:58.812830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:58.812885    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:58.812904    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:58.812938    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:58.812969    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:06.812951882 +0000 UTC m=+50.713257079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:21:58.813401 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:21:58.812995    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:06.812981247 +0000 UTC m=+50.713286449 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:22:00.967610 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.967569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r2j9g" event={"ID":"aa14fe47-a4ae-4b17-8b81-3965e3c2674b","Type":"ContainerStarted","Data":"dc23ca2f62eaa2354d2827a4e76e06abcafc6e282b6d5a248efd30a8449c393e"}
Jul 10 20:22:00.969073 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.969050    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerStarted","Data":"86d4a699c4b16e265cc59df867914d42bfb1ba4cf35b2a766c078e51ac4a8870"}
Jul 10 20:22:00.969194 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.969090    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:22:00.970606 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.970557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerStarted","Data":"704e3e0d4fc3f268db7c24af2db3dcdfbb52a08e22adddb3b988106f02f0c9d9"}
Jul 10 20:22:00.971026 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.971007    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:22:00.972396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.972375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerStarted","Data":"f0136f23410e031d827958ca8835d8359d6e497eee247f4c49005988b77549bb"}
Jul 10 20:22:00.972470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.972405    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerStarted","Data":"d2755c8b80024dfdfa755b00d950360ebcc39b4a5d82f2094ed54eda64662ace"}
Jul 10 20:22:00.972470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.972417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerStarted","Data":"6e6570c6df3ae1e8a5b96b8c234d069970c586bfd23044572ef1fdacb59cd55e"}
Jul 10 20:22:00.984133 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.984100    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r2j9g" podStartSLOduration=2.117101216 podStartE2EDuration="9.984089374s" totalImagesPullingTime="7.866988158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:00.983134773 +0000 UTC m=+44.883440024" watchObservedRunningTime="2026-07-10 20:22:00.984089374 +0000 UTC m=+44.884394570"
Jul 10 20:22:00.998335 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:00.998296    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" podStartSLOduration=16.139337634 podStartE2EDuration="23.998285764s" totalImagesPullingTime="7.85894813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:00.997721459 +0000 UTC m=+44.898026680" watchObservedRunningTime="2026-07-10 20:22:00.998285764 +0000 UTC m=+44.898590961"
Jul 10 20:22:01.013305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:01.013253    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" podStartSLOduration=16.143111599 podStartE2EDuration="24.013236414s" totalImagesPullingTime="7.870124815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:01.012382898 +0000 UTC m=+44.912688116" watchObservedRunningTime="2026-07-10 20:22:01.013236414 +0000 UTC m=+44.913541635"
Jul 10 20:22:01.029893 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:01.029849    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" podStartSLOduration=16.152494943 podStartE2EDuration="24.029834017s" totalImagesPullingTime="7.877339074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:01.02905619 +0000 UTC m=+44.929361400" watchObservedRunningTime="2026-07-10 20:22:01.029834017 +0000 UTC m=+44.930139236"
Jul 10 20:22:06.867871 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:06.867836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:06.867882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:06.867967    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:06.867978    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:06.868019    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:22.86800625 +0000 UTC m=+66.768311446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:06.867966    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:22:06.868231 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:06.868083    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:22.868069542 +0000 UTC m=+66.768374740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:22:16.916873 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:16.916845    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fqfdp"
Jul 10 20:22:22.377244 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.377188    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:22:22.377798 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.377260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:22:22.380071 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.380054    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 10 20:22:22.380124 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.380098    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 10 20:22:22.388380 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.388353    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 20:22:22.388461 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.388419    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:23:26.388399294 +0000 UTC m=+130.288704494 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : secret "metrics-daemon-secret" not found
Jul 10 20:22:22.392116 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.392092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8095de59-42e6-4f38-9280-7761a61ef4ae-original-pull-secret\") pod \"global-pull-secret-syncer-gqc7d\" (UID: \"8095de59-42e6-4f38-9280-7761a61ef4ae\") " pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:22:22.477775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.477739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:22:22.480661 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.480641    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 10 20:22:22.491408 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.491384    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 10 20:22:22.502036 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.502007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktl2m\" (UniqueName: \"kubernetes.io/projected/8768c4f5-0443-4905-9d52-3b7b4ac32e09-kube-api-access-ktl2m\") pod \"network-check-target-ml7ds\" (UID: \"8768c4f5-0443-4905-9d52-3b7b4ac32e09\") " pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:22:22.505787 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.505768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqc7d"
Jul 10 20:22:22.518480 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.518465    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pk8lx\""
Jul 10 20:22:22.526913 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.526887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:22:22.646075 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.646000    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqc7d"]
Jul 10 20:22:22.649593 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:22:22.649566    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8095de59_42e6_4f38_9280_7761a61ef4ae.slice/crio-38b04bac1707a4dede1aba498e8fc4a9cbbd1c491011327a51ba119349613a7e WatchSource:0}: Error finding container 38b04bac1707a4dede1aba498e8fc4a9cbbd1c491011327a51ba119349613a7e: Status 404 returned error can't find the container with id 38b04bac1707a4dede1aba498e8fc4a9cbbd1c491011327a51ba119349613a7e
Jul 10 20:22:22.661605 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.661577    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ml7ds"]
Jul 10 20:22:22.665890 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:22:22.665866    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8768c4f5_0443_4905_9d52_3b7b4ac32e09.slice/crio-b2ce0f02a1608f4a3d3ca109441c65edab2076a62d9acdc3130e6222548c41cd WatchSource:0}: Error finding container b2ce0f02a1608f4a3d3ca109441c65edab2076a62d9acdc3130e6222548c41cd: Status 404 returned error can't find the container with id b2ce0f02a1608f4a3d3ca109441c65edab2076a62d9acdc3130e6222548c41cd
Jul 10 20:22:22.881636 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.881595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:22:22.881803 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:22.881712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:22:22.881803 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.881742    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:22:22.881803 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.881762    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:22:22.881968 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.881823    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:54.881807333 +0000 UTC m=+98.782112533 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:22:22.881968 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.881840    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:22:22.881968 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:22.881890    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:22:54.881874629 +0000 UTC m=+98.782179829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:22:23.037288 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:23.037206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqc7d" event={"ID":"8095de59-42e6-4f38-9280-7761a61ef4ae","Type":"ContainerStarted","Data":"38b04bac1707a4dede1aba498e8fc4a9cbbd1c491011327a51ba119349613a7e"}
Jul 10 20:22:23.038296 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:23.038261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ml7ds" event={"ID":"8768c4f5-0443-4905-9d52-3b7b4ac32e09","Type":"ContainerStarted","Data":"b2ce0f02a1608f4a3d3ca109441c65edab2076a62d9acdc3130e6222548c41cd"}
Jul 10 20:22:28.056913 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:28.056878    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ml7ds" event={"ID":"8768c4f5-0443-4905-9d52-3b7b4ac32e09","Type":"ContainerStarted","Data":"3097ec80707a5a344067702b0aa4c9762e816f8df771a351479bf8b1d814b7d9"}
Jul 10 20:22:28.057402 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:28.056964    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:22:28.058351 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:28.058330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqc7d" event={"ID":"8095de59-42e6-4f38-9280-7761a61ef4ae","Type":"ContainerStarted","Data":"e53cae12dce96bc679014b67da83cec31273a7f3f8b1177e76a4d1aa7aa1d600"}
Jul 10 20:22:28.073084 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:28.073041    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ml7ds" podStartSLOduration=67.692479203 podStartE2EDuration="1m12.073026351s" totalImagesPullingTime="4.380547148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:28.071928305 +0000 UTC m=+71.972233539" watchObservedRunningTime="2026-07-10 20:22:28.073026351 +0000 UTC m=+71.973331569"
Jul 10 20:22:28.087526 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:28.087466    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gqc7d" podStartSLOduration=66.685420192 podStartE2EDuration="1m11.087453138s" totalImagesPullingTime="4.402032946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:22:28.08716021 +0000 UTC m=+71.987465428" watchObservedRunningTime="2026-07-10 20:22:28.087453138 +0000 UTC m=+71.987758356"
Jul 10 20:22:54.918093 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:54.918054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:54.918132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:54.918205    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:54.918225    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:54.918256    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:54.918306    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:23:58.918288205 +0000 UTC m=+162.818593418 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:22:54.918563 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:22:54.918320    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:23:58.918313899 +0000 UTC m=+162.818619095 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:22:59.067288 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:22:59.067253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ml7ds"
Jul 10 20:23:26.437897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:23:26.437857    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:23:26.438356 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:26.437994    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 10 20:23:26.438356 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:26.438061    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs podName:2b72c49e-6395-4d9f-9495-d52457c4680c nodeName:}" failed. No retries permitted until 2026-07-10 20:25:28.438045154 +0000 UTC m=+252.338350351 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs") pod "network-metrics-daemon-cfp74" (UID: "2b72c49e-6395-4d9f-9495-d52457c4680c") : secret "metrics-daemon-secret" not found
Jul 10 20:23:54.018133 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:54.018086    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-656b66b887-9w49z" podUID="b635d89e-3223-4884-940e-36f72629a910"
Jul 10 20:23:54.078738 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:54.078693    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nc87d" podUID="58873ee7-32c0-4d3b-a18a-3ab6fd149297"
Jul 10 20:23:54.310408 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:23:54.310323    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:23:54.310569 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:23:54.310327    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:23:55.824213 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:55.824167    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cfp74" podUID="2b72c49e-6395-4d9f-9495-d52457c4680c"
Jul 10 20:23:58.950553 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:23:58.950510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:23:58.950575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:58.950658    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:58.950713    2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:58.950730    2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-656b66b887-9w49z: secret "image-registry-tls" not found
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:58.950750    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls podName:58873ee7-32c0-4d3b-a18a-3ab6fd149297 nodeName:}" failed. No retries permitted until 2026-07-10 20:26:00.950734277 +0000 UTC m=+284.851039474 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls") pod "dns-default-nc87d" (UID: "58873ee7-32c0-4d3b-a18a-3ab6fd149297") : secret "dns-default-metrics-tls" not found
Jul 10 20:23:58.950948 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:23:58.950778    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls podName:b635d89e-3223-4884-940e-36f72629a910 nodeName:}" failed. No retries permitted until 2026-07-10 20:26:00.950766686 +0000 UTC m=+284.851071886 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls") pod "image-registry-656b66b887-9w49z" (UID: "b635d89e-3223-4884-940e-36f72629a910") : secret "image-registry-tls" not found
Jul 10 20:24:00.327250 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.327214    2564 generic.go:356] "Generic (PLEG): container finished" podID="856a7f25-761a-4874-bf5f-1be0594090e2" containerID="86d4a699c4b16e265cc59df867914d42bfb1ba4cf35b2a766c078e51ac4a8870" exitCode=1
Jul 10 20:24:00.327708 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.327282    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerDied","Data":"86d4a699c4b16e265cc59df867914d42bfb1ba4cf35b2a766c078e51ac4a8870"}
Jul 10 20:24:00.327708 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.327629    2564 scope.go:117] "RemoveContainer" containerID="86d4a699c4b16e265cc59df867914d42bfb1ba4cf35b2a766c078e51ac4a8870"
Jul 10 20:24:00.328851 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.328753    2564 generic.go:356] "Generic (PLEG): container finished" podID="4af0945f-e5ce-416e-90dd-2441b84f8afd" containerID="704e3e0d4fc3f268db7c24af2db3dcdfbb52a08e22adddb3b988106f02f0c9d9" exitCode=255
Jul 10 20:24:00.328851 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.328806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerDied","Data":"704e3e0d4fc3f268db7c24af2db3dcdfbb52a08e22adddb3b988106f02f0c9d9"}
Jul 10 20:24:00.329088 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.329073    2564 scope.go:117] "RemoveContainer" containerID="704e3e0d4fc3f268db7c24af2db3dcdfbb52a08e22adddb3b988106f02f0c9d9"
Jul 10 20:24:00.969395 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:00.969345    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:24:01.329632 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:01.329610    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:24:01.333482 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:01.333455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerStarted","Data":"749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"}
Jul 10 20:24:01.333682 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:01.333663    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:24:01.334236 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:01.334210    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:24:01.335181 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:01.335161    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerStarted","Data":"baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91"}
Jul 10 20:24:08.778816 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:24:08.778737    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:25:07.438253 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.438224    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-645jh"]
Jul 10 20:25:07.441081 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.441065    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.450211 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.450185    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Jul 10 20:25:07.451091 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.451064    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 10 20:25:07.451091 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.451065    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 10 20:25:07.456052 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.456029    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-mrs5z\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrs5z\"" type="*v1.Secret"
Jul 10 20:25:07.456304 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.456287    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 10 20:25:07.456348 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:07.456302    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-645jh\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="ea90649f-2f85-417d-afde-75123e21ad70" pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.471783 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.471761    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-645jh"]
Jul 10 20:25:07.524097 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.524073    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.524189 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.524142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea90649f-2f85-417d-afde-75123e21ad70-data-volume\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.524189 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.524184    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea90649f-2f85-417d-afde-75123e21ad70-crio-socket\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.524257 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.524234    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.524290 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.524258    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625331 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625426 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625426 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625512 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625469    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea90649f-2f85-417d-afde-75123e21ad70-data-volume\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea90649f-2f85-417d-afde-75123e21ad70-crio-socket\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625638 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625624    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ea90649f-2f85-417d-afde-75123e21ad70-crio-socket\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:07.625760 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:07.625746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ea90649f-2f85-417d-afde-75123e21ad70-data-volume\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:08.571575 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:08.571548    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 10 20:25:08.575972 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.575953    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 20:25:08.576063 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.576023    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:09.076009442 +0000 UTC m=+232.976314674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : configmap "kube-rbac-proxy" not found
Jul 10 20:25:08.626380 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.626364    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Jul 10 20:25:08.626428 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.626420    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:09.126407492 +0000 UTC m=+233.026712692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : failed to sync secret cache: timed out waiting for the condition
Jul 10 20:25:08.632939 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.632917    2564 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 10 20:25:08.714545 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:08.714526    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrs5z\""
Jul 10 20:25:08.808099 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:08.808080    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 10 20:25:08.868410 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:08.868369    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 10 20:25:08.873595 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.873583    2564 projected.go:196] Error preparing data for projected volume kube-api-access-4gg85 for pod openshift-insights/insights-runtime-extractor-645jh: failed to sync configmap cache: timed out waiting for the condition
Jul 10 20:25:08.873669 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:08.873659    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85 podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:09.373642933 +0000 UTC m=+233.273948148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4gg85" (UniqueName: "kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : failed to sync configmap cache: timed out waiting for the condition
Jul 10 20:25:08.904805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:08.904790    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 10 20:25:09.137711 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:09.137649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:09.137711 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:09.137684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:09.137842 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:09.137753    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 20:25:09.137842 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:09.137789    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:09.137842 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:09.137802    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:10.137789879 +0000 UTC m=+234.038095079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : configmap "kube-rbac-proxy" not found
Jul 10 20:25:09.137842 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:09.137826    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:10.137814253 +0000 UTC m=+234.038119451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:09.439714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:09.439664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:09.441839 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:09.441814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gg85\" (UniqueName: \"kubernetes.io/projected/ea90649f-2f85-417d-afde-75123e21ad70-kube-api-access-4gg85\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:09.723106 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:09.723050    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjqwh_395ceee6-e47f-4442-8897-6b4a0075f68b/dns-node-resolver/0.log"
Jul 10 20:25:10.144227 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:10.144203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:10.144339 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:10.144240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:10.144339 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:10.144311    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 20:25:10.144405 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:10.144356    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:10.144405 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:10.144372    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:12.144357849 +0000 UTC m=+236.044663047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : configmap "kube-rbac-proxy" not found
Jul 10 20:25:10.144405 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:10.144394    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:12.144382453 +0000 UTC m=+236.044687649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:10.522656 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:10.522591    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnq7j_b5d0e425-31ca-4ab8-bd99-8534f583d901/node-ca/0.log"
Jul 10 20:25:11.323386 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:11.323344    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2j9g_aa14fe47-a4ae-4b17-8b81-3965e3c2674b/serve-healthcheck-canary/0.log"
Jul 10 20:25:12.158262 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:12.158203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:12.158262 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:12.158268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:12.158578 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:12.158342    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 20:25:12.158578 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:12.158396    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:12.158578 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:12.158435    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:16.158413239 +0000 UTC m=+240.058718448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : configmap "kube-rbac-proxy" not found
Jul 10 20:25:12.158578 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:12.158454    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:16.158445768 +0000 UTC m=+240.058750972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:16.188563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:16.188515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:16.188563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:16.188571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:16.189089 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:16.188662    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 10 20:25:16.189089 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:16.188677    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:16.189089 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:16.188764    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:24.18874164 +0000 UTC m=+248.089046860 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : configmap "kube-rbac-proxy" not found
Jul 10 20:25:16.189089 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:16.188818    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls podName:ea90649f-2f85-417d-afde-75123e21ad70 nodeName:}" failed. No retries permitted until 2026-07-10 20:25:24.188806597 +0000 UTC m=+248.089111795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls") pod "insights-runtime-extractor-645jh" (UID: "ea90649f-2f85-417d-afde-75123e21ad70") : secret "insights-runtime-extractor-tls" not found
Jul 10 20:25:24.246921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.246889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:24.247272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.246932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:24.247439 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.247410    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ea90649f-2f85-417d-afde-75123e21ad70-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:24.249184 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.249166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ea90649f-2f85-417d-afde-75123e21ad70-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-645jh\" (UID: \"ea90649f-2f85-417d-afde-75123e21ad70\") " pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:24.251906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.251891    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mrs5z\""
Jul 10 20:25:24.260070 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.260054    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-645jh"
Jul 10 20:25:24.377682 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.377654    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-645jh"]
Jul 10 20:25:24.379938 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:25:24.379912    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea90649f_2f85_417d_afde_75123e21ad70.slice/crio-6a593105b40709d0143909d0a9f62be171d55dd6573bf9e56f1d4b231867bc75 WatchSource:0}: Error finding container 6a593105b40709d0143909d0a9f62be171d55dd6573bf9e56f1d4b231867bc75: Status 404 returned error can't find the container with id 6a593105b40709d0143909d0a9f62be171d55dd6573bf9e56f1d4b231867bc75
Jul 10 20:25:24.563470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:24.563440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-645jh" event={"ID":"ea90649f-2f85-417d-afde-75123e21ad70","Type":"ContainerStarted","Data":"6a593105b40709d0143909d0a9f62be171d55dd6573bf9e56f1d4b231867bc75"}
Jul 10 20:25:28.484962 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:28.484597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:25:28.487410 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:28.487378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b72c49e-6395-4d9f-9495-d52457c4680c-metrics-certs\") pod \"network-metrics-daemon-cfp74\" (UID: \"2b72c49e-6395-4d9f-9495-d52457c4680c\") " pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:25:28.582054 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:28.582018    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr56z\""
Jul 10 20:25:28.589560 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:28.589531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfp74"
Jul 10 20:25:28.731148 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:28.731114    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfp74"]
Jul 10 20:25:28.734550 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:25:28.734520    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b72c49e_6395_4d9f_9495_d52457c4680c.slice/crio-8069a0b2fadd44184a387229bbe343d136a23bf7921a28c12062ae6d2e1d267a WatchSource:0}: Error finding container 8069a0b2fadd44184a387229bbe343d136a23bf7921a28c12062ae6d2e1d267a: Status 404 returned error can't find the container with id 8069a0b2fadd44184a387229bbe343d136a23bf7921a28c12062ae6d2e1d267a
Jul 10 20:25:29.580748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:29.580707    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfp74" event={"ID":"2b72c49e-6395-4d9f-9495-d52457c4680c","Type":"ContainerStarted","Data":"8069a0b2fadd44184a387229bbe343d136a23bf7921a28c12062ae6d2e1d267a"}
Jul 10 20:25:31.589175 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:31.589138    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfp74" event={"ID":"2b72c49e-6395-4d9f-9495-d52457c4680c","Type":"ContainerStarted","Data":"fd2ae87161d98c41b08234277472a7851f99951068938f9c5c2b14934307c10e"}
Jul 10 20:25:31.589587 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:31.589201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfp74" event={"ID":"2b72c49e-6395-4d9f-9495-d52457c4680c","Type":"ContainerStarted","Data":"72ba745c2ab29481c4efa23ecec94713d40dfca607ccd905f6db6e20b00bee72"}
Jul 10 20:25:31.590552 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:31.590531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-645jh" event={"ID":"ea90649f-2f85-417d-afde-75123e21ad70","Type":"ContainerStarted","Data":"661d88a3c1a7105767cba2c209e7af95bfc9f7bb0dfbfd7d42a427afeec1c75b"}
Jul 10 20:25:32.595316 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:32.595283    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-645jh" event={"ID":"ea90649f-2f85-417d-afde-75123e21ad70","Type":"ContainerStarted","Data":"149efc52b66305e260896234e07d53544ebd4edf2ac87dbb21c1f9106245a1b3"}
Jul 10 20:25:35.606499 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:35.606452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-645jh" event={"ID":"ea90649f-2f85-417d-afde-75123e21ad70","Type":"ContainerStarted","Data":"36cf68a472c955be8b8aa2c685157d0e5f84d5bd88949662aba1cac684be85cd"}
Jul 10 20:25:35.624078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:35.624015    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-645jh" podStartSLOduration=18.402709665 podStartE2EDuration="28.624001929s" totalImagesPullingTime="10.221292264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 20:25:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:25:35.62266412 +0000 UTC m=+259.522969339" watchObservedRunningTime="2026-07-10 20:25:35.624001929 +0000 UTC m=+259.524307149"
Jul 10 20:25:35.624396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:35.624376    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cfp74" podStartSLOduration=256.952649827 podStartE2EDuration="4m19.624369257s" totalImagesPullingTime="2.67171943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:25:31.607080419 +0000 UTC m=+255.507385650" watchObservedRunningTime="2026-07-10 20:25:35.624369257 +0000 UTC m=+259.524674476"
Jul 10 20:25:56.475891 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.475810    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p8gxp"]
Jul 10 20:25:56.493537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.493513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.496556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496526    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 10 20:25:56.496708 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496526    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 10 20:25:56.496787 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496566    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 10 20:25:56.496787 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496566    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 10 20:25:56.496900 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496627    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 10 20:25:56.496900 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.496667    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 10 20:25:56.497697 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.497682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d9kfq\""
Jul 10 20:25:56.577704 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-metrics-client-ca\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577807 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577807 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577741    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-root\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzfn\" (UniqueName: \"kubernetes.io/projected/8d545587-46d7-4c0b-b6d5-48054c46debe-kube-api-access-4hzfn\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577844    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-wtmp\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-textfile\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-sys\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.577977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.577921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.678952 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.678929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679086 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.678974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-metrics-client-ca\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679086 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679086 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-root\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679086 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzfn\" (UniqueName: \"kubernetes.io/projected/8d545587-46d7-4c0b-b6d5-48054c46debe-kube-api-access-4hzfn\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679086 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679076    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-wtmp\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-textfile\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-root\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679152    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-sys\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-sys\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679305 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:56.679284    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 10 20:25:56.679595 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:56.679344    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls podName:8d545587-46d7-4c0b-b6d5-48054c46debe nodeName:}" failed. No retries permitted until 2026-07-10 20:25:57.179326693 +0000 UTC m=+281.079631891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls") pod "node-exporter-p8gxp" (UID: "8d545587-46d7-4c0b-b6d5-48054c46debe") : secret "node-exporter-tls" not found
Jul 10 20:25:56.679595 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-wtmp\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679688 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-textfile\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679739 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-accelerators-collector-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.679773 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.679723    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d545587-46d7-4c0b-b6d5-48054c46debe-metrics-client-ca\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.681369 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.681351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:56.690560 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:56.690542    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzfn\" (UniqueName: \"kubernetes.io/projected/8d545587-46d7-4c0b-b6d5-48054c46debe-kube-api-access-4hzfn\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:57.182613 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.182585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:57.184648 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.184632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d545587-46d7-4c0b-b6d5-48054c46debe-node-exporter-tls\") pod \"node-exporter-p8gxp\" (UID: \"8d545587-46d7-4c0b-b6d5-48054c46debe\") " pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:57.311031 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:57.310982    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-656b66b887-9w49z" podUID="b635d89e-3223-4884-940e-36f72629a910"
Jul 10 20:25:57.311031 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:25:57.310991    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nc87d" podUID="58873ee7-32c0-4d3b-a18a-3ab6fd149297"
Jul 10 20:25:57.402485 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.402458    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p8gxp"
Jul 10 20:25:57.411347 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:25:57.411327    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d545587_46d7_4c0b_b6d5_48054c46debe.slice/crio-dd9fb60d6a0b9786b67487afcbfe0af770cc7a92920e4dab3c0fa6dd3e774c98 WatchSource:0}: Error finding container dd9fb60d6a0b9786b67487afcbfe0af770cc7a92920e4dab3c0fa6dd3e774c98: Status 404 returned error can't find the container with id dd9fb60d6a0b9786b67487afcbfe0af770cc7a92920e4dab3c0fa6dd3e774c98
Jul 10 20:25:57.668784 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.668751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8gxp" event={"ID":"8d545587-46d7-4c0b-b6d5-48054c46debe","Type":"ContainerStarted","Data":"dd9fb60d6a0b9786b67487afcbfe0af770cc7a92920e4dab3c0fa6dd3e774c98"}
Jul 10 20:25:57.669111 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.668787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:25:57.669111 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:57.668787    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:25:59.676063 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:59.676026    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d545587-46d7-4c0b-b6d5-48054c46debe" containerID="9e785a26a4a411aaecf05a828ff05229f74501a8deaafb339642bca1023a60e1" exitCode=0
Jul 10 20:25:59.676434 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:25:59.676118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8gxp" event={"ID":"8d545587-46d7-4c0b-b6d5-48054c46debe","Type":"ContainerDied","Data":"9e785a26a4a411aaecf05a828ff05229f74501a8deaafb339642bca1023a60e1"}
Jul 10 20:26:00.681690 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.681657    2564 generic.go:356] "Generic (PLEG): container finished" podID="856a7f25-761a-4874-bf5f-1be0594090e2" containerID="749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420" exitCode=1
Jul 10 20:26:00.682115 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.681728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerDied","Data":"749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"}
Jul 10 20:26:00.682115 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.681782    2564 scope.go:117] "RemoveContainer" containerID="86d4a699c4b16e265cc59df867914d42bfb1ba4cf35b2a766c078e51ac4a8870"
Jul 10 20:26:00.682115 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.682090    2564 scope.go:117] "RemoveContainer" containerID="749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"
Jul 10 20:26:00.682305 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:00.682286    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-578d876cdc-4px4r_open-cluster-management-agent-addon(856a7f25-761a-4874-bf5f-1be0594090e2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" podUID="856a7f25-761a-4874-bf5f-1be0594090e2"
Jul 10 20:26:00.683633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.683615    2564 generic.go:356] "Generic (PLEG): container finished" podID="4af0945f-e5ce-416e-90dd-2441b84f8afd" containerID="baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91" exitCode=255
Jul 10 20:26:00.683711 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.683684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerDied","Data":"baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91"}
Jul 10 20:26:00.683944 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.683927    2564 scope.go:117] "RemoveContainer" containerID="baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91"
Jul 10 20:26:00.684088 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:00.684068    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6cbff74488-cx4mm_open-cluster-management-agent-addon(4af0945f-e5ce-416e-90dd-2441b84f8afd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" podUID="4af0945f-e5ce-416e-90dd-2441b84f8afd"
Jul 10 20:26:00.685729 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.685709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8gxp" event={"ID":"8d545587-46d7-4c0b-b6d5-48054c46debe","Type":"ContainerStarted","Data":"435eb693e968c019bff5b6b21d11ce8bfca7c3f9244e0028c46f55cef5ab92bc"}
Jul 10 20:26:00.685817 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.685749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p8gxp" event={"ID":"8d545587-46d7-4c0b-b6d5-48054c46debe","Type":"ContainerStarted","Data":"cd0670da30aa0bdd6380dbf177c63bcc95041da2551c139101625c43192c9de0"}
Jul 10 20:26:00.691078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.691061    2564 scope.go:117] "RemoveContainer" containerID="704e3e0d4fc3f268db7c24af2db3dcdfbb52a08e22adddb3b988106f02f0c9d9"
Jul 10 20:26:00.730726 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:00.730682    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p8gxp" podStartSLOduration=2.949962909 podStartE2EDuration="4.730670381s" totalImagesPullingTime="1.717584748s" totalInitContainerRuntime="63.122724ms" isStatefulPod=true podCreationTimestamp="2026-07-10 20:25:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:26:00.729814256 +0000 UTC m=+284.630119489" watchObservedRunningTime="2026-07-10 20:26:00.730670381 +0000 UTC m=+284.630975600"
Jul 10 20:26:01.011576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.011472    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:01.011576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.011534    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:26:01.013759 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.013733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58873ee7-32c0-4d3b-a18a-3ab6fd149297-metrics-tls\") pod \"dns-default-nc87d\" (UID: \"58873ee7-32c0-4d3b-a18a-3ab6fd149297\") " pod="openshift-dns/dns-default-nc87d"
Jul 10 20:26:01.013874 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.013857    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"image-registry-656b66b887-9w49z\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") " pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:01.272510 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.272417    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s5ppj\""
Jul 10 20:26:01.272664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.272551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgw85\""
Jul 10 20:26:01.280136 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.280114    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:01.280224 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.280208    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:26:01.330021 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.329985    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:26:01.334693 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.334558    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:26:01.345204 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.345145    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm"
Jul 10 20:26:01.414252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.414131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nc87d"]
Jul 10 20:26:01.416355 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:26:01.416326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58873ee7_32c0_4d3b_a18a_3ab6fd149297.slice/crio-b18079d4a2451e72295a86eed03035128498de2af9e38182c86c5cf9b24f9624 WatchSource:0}: Error finding container b18079d4a2451e72295a86eed03035128498de2af9e38182c86c5cf9b24f9624: Status 404 returned error can't find the container with id b18079d4a2451e72295a86eed03035128498de2af9e38182c86c5cf9b24f9624
Jul 10 20:26:01.433171 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.433153    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:26:01.435208 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:26:01.435171    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb635d89e_3223_4884_940e_36f72629a910.slice/crio-37127220fe96e7d5116688e50cc0026637aa7de44d0647c6884beca667767762 WatchSource:0}: Error finding container 37127220fe96e7d5116688e50cc0026637aa7de44d0647c6884beca667767762: Status 404 returned error can't find the container with id 37127220fe96e7d5116688e50cc0026637aa7de44d0647c6884beca667767762
Jul 10 20:26:01.695921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.695883    2564 scope.go:117] "RemoveContainer" containerID="baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91"
Jul 10 20:26:01.696443 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:01.696072    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6cbff74488-cx4mm_open-cluster-management-agent-addon(4af0945f-e5ce-416e-90dd-2441b84f8afd)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" podUID="4af0945f-e5ce-416e-90dd-2441b84f8afd"
Jul 10 20:26:01.696888 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.696866    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc87d" event={"ID":"58873ee7-32c0-4d3b-a18a-3ab6fd149297","Type":"ContainerStarted","Data":"b18079d4a2451e72295a86eed03035128498de2af9e38182c86c5cf9b24f9624"}
Jul 10 20:26:01.698561 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.698544    2564 scope.go:117] "RemoveContainer" containerID="749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"
Jul 10 20:26:01.698717 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:01.698698    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-578d876cdc-4px4r_open-cluster-management-agent-addon(856a7f25-761a-4874-bf5f-1be0594090e2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" podUID="856a7f25-761a-4874-bf5f-1be0594090e2"
Jul 10 20:26:01.699764 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.699742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-656b66b887-9w49z" event={"ID":"b635d89e-3223-4884-940e-36f72629a910","Type":"ContainerStarted","Data":"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"}
Jul 10 20:26:01.699862 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.699787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-656b66b887-9w49z" event={"ID":"b635d89e-3223-4884-940e-36f72629a910","Type":"ContainerStarted","Data":"37127220fe96e7d5116688e50cc0026637aa7de44d0647c6884beca667767762"}
Jul 10 20:26:01.700039 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.700022    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:01.756236 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:01.756171    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-656b66b887-9w49z" podStartSLOduration=284.75615064 podStartE2EDuration="4m44.75615064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:26:01.754809857 +0000 UTC m=+285.655115076" watchObservedRunningTime="2026-07-10 20:26:01.75615064 +0000 UTC m=+285.656455860"
Jul 10 20:26:02.703819 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:02.703785    2564 scope.go:117] "RemoveContainer" containerID="749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"
Jul 10 20:26:02.704261 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:02.704000    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-578d876cdc-4px4r_open-cluster-management-agent-addon(856a7f25-761a-4874-bf5f-1be0594090e2)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" podUID="856a7f25-761a-4874-bf5f-1be0594090e2"
Jul 10 20:26:03.708155 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:03.708120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc87d" event={"ID":"58873ee7-32c0-4d3b-a18a-3ab6fd149297","Type":"ContainerStarted","Data":"6c2031b4013213259a1ee2dfccb3340ae8911bcc56aa2ff8973d3c4076380310"}
Jul 10 20:26:03.708583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:03.708173    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc87d" event={"ID":"58873ee7-32c0-4d3b-a18a-3ab6fd149297","Type":"ContainerStarted","Data":"3b11c4a0e93c39a6d40d10d0b19d7046738964afcb3507f08142bb70f14587c7"}
Jul 10 20:26:03.708583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:03.708263    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:26:03.725585 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:03.725532    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nc87d" podStartSLOduration=250.835264749 podStartE2EDuration="4m12.725515466s" totalImagesPullingTime="1.890250717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:26:03.72415313 +0000 UTC m=+287.624458380" watchObservedRunningTime="2026-07-10 20:26:03.725515466 +0000 UTC m=+287.625820692"
Jul 10 20:26:11.353039 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:11.352991    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" podUID="190eea48-564e-4b13-bc2e-9dec8aba9dd7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 20:26:13.713815 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:13.713787    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nc87d"
Jul 10 20:26:16.603985 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:16.603960    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:26:16.604407 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:16.604046    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:26:16.606601 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:16.606578    2564 kubelet.go:1648] "Image garbage collection succeeded"
Jul 10 20:26:16.784231 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:16.784122    2564 scope.go:117] "RemoveContainer" containerID="baaaf33c51190095d03e337a53cb2901b7ef88282b81a595acdabfa9d23b8a91"
Jul 10 20:26:17.755776 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:17.755740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6cbff74488-cx4mm" event={"ID":"4af0945f-e5ce-416e-90dd-2441b84f8afd","Type":"ContainerStarted","Data":"a16d9d3dd5853f40c7c64260351d1ef44951297c3373b8d1913969550ecf17ac"}
Jul 10 20:26:17.778636 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:17.778609    2564 scope.go:117] "RemoveContainer" containerID="749abb70607f33b6c81ff59a71e366cb8771eae35cc9b835ea4a8353e497b420"
Jul 10 20:26:18.760797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:18.760765    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r" event={"ID":"856a7f25-761a-4874-bf5f-1be0594090e2","Type":"ContainerStarted","Data":"df9ecf011a0b04ae16813d4be92b0868cb84fa89eb37af7319392571955a021e"}
Jul 10 20:26:18.761199 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:18.761055    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:26:18.762435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:18.762411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578d876cdc-4px4r"
Jul 10 20:26:21.284869 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:21.284834    2564 patch_prober.go:28] interesting pod/image-registry-656b66b887-9w49z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 10 20:26:21.285240 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:21.284883    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-656b66b887-9w49z" podUID="b635d89e-3223-4884-940e-36f72629a910" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 10 20:26:21.352872 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:21.352837    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" podUID="190eea48-564e-4b13-bc2e-9dec8aba9dd7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 20:26:22.708382 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:22.708357    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:30.316437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:30.316403    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:26:31.353377 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.353336    2564 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" podUID="190eea48-564e-4b13-bc2e-9dec8aba9dd7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 10 20:26:31.353758 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.353415    2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd"
Jul 10 20:26:31.353910 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.353880    2564 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"f0136f23410e031d827958ca8835d8359d6e497eee247f4c49005988b77549bb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 10 20:26:31.353947 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.353929    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" podUID="190eea48-564e-4b13-bc2e-9dec8aba9dd7" containerName="service-proxy" containerID="cri-o://f0136f23410e031d827958ca8835d8359d6e497eee247f4c49005988b77549bb" gracePeriod=30
Jul 10 20:26:31.808142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.808065    2564 generic.go:356] "Generic (PLEG): container finished" podID="190eea48-564e-4b13-bc2e-9dec8aba9dd7" containerID="f0136f23410e031d827958ca8835d8359d6e497eee247f4c49005988b77549bb" exitCode=2
Jul 10 20:26:31.808142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.808107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerDied","Data":"f0136f23410e031d827958ca8835d8359d6e497eee247f4c49005988b77549bb"}
Jul 10 20:26:31.808142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:31.808137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-54bb96946d-zfbtd" event={"ID":"190eea48-564e-4b13-bc2e-9dec8aba9dd7","Type":"ContainerStarted","Data":"e66dfc045b45dd78cfa6330d61ef013088eb3b9b7bd26b597abde4eac574eaa7"}
Jul 10 20:26:37.977009 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:37.976962    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-656b66b887-9w49z_b635d89e-3223-4884-940e-36f72629a910/registry/0.log"
Jul 10 20:26:38.375191 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:38.375166    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnq7j_b5d0e425-31ca-4ab8-bd99-8534f583d901/node-ca/0.log"
Jul 10 20:26:39.177396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:39.177364    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2j9g_aa14fe47-a4ae-4b17-8b81-3965e3c2674b/serve-healthcheck-canary/0.log"
Jul 10 20:26:55.335539 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:55.335460    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-656b66b887-9w49z" podUID="b635d89e-3223-4884-940e-36f72629a910" containerName="registry" containerID="cri-o://012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6" gracePeriod=30
Jul 10 20:26:56.576295 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.576271    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:56.692973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.692889    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.692973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.692934    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.692973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.692957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.692973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.692974    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.693280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.692992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.693280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.693021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.693280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.693053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg6df\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.693280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.693086    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca\") pod \"b635d89e-3223-4884-940e-36f72629a910\" (UID: \"b635d89e-3223-4884-940e-36f72629a910\") "
Jul 10 20:26:56.693479 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.693337    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:26:56.693629 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.693603    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:26:56.695698 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.695666    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:26:56.695852 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.695831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:26:56.695912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.695864    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df" (OuterVolumeSpecName: "kube-api-access-xg6df") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "kube-api-access-xg6df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:26:56.695949 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.695922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:26:56.696073 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.696050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:26:56.701726 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.701704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b635d89e-3223-4884-940e-36f72629a910" (UID: "b635d89e-3223-4884-940e-36f72629a910"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:26:56.793811 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793772    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6df\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-kube-api-access-xg6df\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.793811 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793807    2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-trusted-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.793811 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793821    2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-registry-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.794006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793833    2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-installation-pull-secrets\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.794006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793846    2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b635d89e-3223-4884-940e-36f72629a910-registry-certificates\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.794006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793858    2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b635d89e-3223-4884-940e-36f72629a910-ca-trust-extracted\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.794006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793870    2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b635d89e-3223-4884-940e-36f72629a910-image-registry-private-configuration\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.794006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.793882    2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b635d89e-3223-4884-940e-36f72629a910-bound-sa-token\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:26:56.889896 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.889862    2564 generic.go:356] "Generic (PLEG): container finished" podID="b635d89e-3223-4884-940e-36f72629a910" containerID="012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6" exitCode=0
Jul 10 20:26:56.890035 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.889900    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-656b66b887-9w49z"
Jul 10 20:26:56.890035 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.889937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-656b66b887-9w49z" event={"ID":"b635d89e-3223-4884-940e-36f72629a910","Type":"ContainerDied","Data":"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"}
Jul 10 20:26:56.890035 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.889980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-656b66b887-9w49z" event={"ID":"b635d89e-3223-4884-940e-36f72629a910","Type":"ContainerDied","Data":"37127220fe96e7d5116688e50cc0026637aa7de44d0647c6884beca667767762"}
Jul 10 20:26:56.890035 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.890001    2564 scope.go:117] "RemoveContainer" containerID="012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"
Jul 10 20:26:56.897824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.897810    2564 scope.go:117] "RemoveContainer" containerID="012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"
Jul 10 20:26:56.898094 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:26:56.898076    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6\": container with ID starting with 012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6 not found: ID does not exist" containerID="012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"
Jul 10 20:26:56.898139 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.898104    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6"} err="failed to get container status \"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6\": rpc error: code = NotFound desc = could not find container \"012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6\": container with ID starting with 012409b816bada15c6ed3a1a17bf55a5b994e6058447de8c726930d5fa50fae6 not found: ID does not exist"
Jul 10 20:26:56.905366 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.905346    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:26:56.912458 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:56.909853    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-656b66b887-9w49z"]
Jul 10 20:26:58.783603 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:26:58.783565    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b635d89e-3223-4884-940e-36f72629a910" path="/var/lib/kubelet/pods/b635d89e-3223-4884-940e-36f72629a910/volumes"
Jul 10 20:29:05.550470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.550435    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"]
Jul 10 20:29:05.550926 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.550675    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b635d89e-3223-4884-940e-36f72629a910" containerName="registry"
Jul 10 20:29:05.550926 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.550685    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b635d89e-3223-4884-940e-36f72629a910" containerName="registry"
Jul 10 20:29:05.550926 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.550739    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b635d89e-3223-4884-940e-36f72629a910" containerName="registry"
Jul 10 20:29:05.554459 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.554424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.557322 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.557300    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 10 20:29:05.557454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.557352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 10 20:29:05.558348 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.558332    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xhxnx\""
Jul 10 20:29:05.564395 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.564377    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"]
Jul 10 20:29:05.737461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.737438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7745869-52e5-45bf-abbd-bf751dd3174f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.737584 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.737470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lb7\" (UniqueName: \"kubernetes.io/projected/f7745869-52e5-45bf-abbd-bf751dd3174f-kube-api-access-j6lb7\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.838749 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.838724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7745869-52e5-45bf-abbd-bf751dd3174f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.838864 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.838755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lb7\" (UniqueName: \"kubernetes.io/projected/f7745869-52e5-45bf-abbd-bf751dd3174f-kube-api-access-j6lb7\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.839085 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.839067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7745869-52e5-45bf-abbd-bf751dd3174f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.847232 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.847205    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lb7\" (UniqueName: \"kubernetes.io/projected/f7745869-52e5-45bf-abbd-bf751dd3174f-kube-api-access-j6lb7\") pod \"openshift-lws-operator-fd8ccff4c-snhzk\" (UID: \"f7745869-52e5-45bf-abbd-bf751dd3174f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.863167 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.863140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"
Jul 10 20:29:05.980695 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.980666    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk"]
Jul 10 20:29:05.983512 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:05.983471    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7745869_52e5_45bf_abbd_bf751dd3174f.slice/crio-d50384f541d5729acb2e372b691bdb2dd22ba04b99f18c8985b419ce0d5f947a WatchSource:0}: Error finding container d50384f541d5729acb2e372b691bdb2dd22ba04b99f18c8985b419ce0d5f947a: Status 404 returned error can't find the container with id d50384f541d5729acb2e372b691bdb2dd22ba04b99f18c8985b419ce0d5f947a
Jul 10 20:29:05.984912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:05.984895    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:29:06.297013 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:06.296955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk" event={"ID":"f7745869-52e5-45bf-abbd-bf751dd3174f","Type":"ContainerStarted","Data":"d50384f541d5729acb2e372b691bdb2dd22ba04b99f18c8985b419ce0d5f947a"}
Jul 10 20:29:10.312563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:10.312527    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk" event={"ID":"f7745869-52e5-45bf-abbd-bf751dd3174f","Type":"ContainerStarted","Data":"27aed4ceb12fc443d3c47b352a0098d0e386b329b38aad98a6ae83ed71dd798c"}
Jul 10 20:29:10.328925 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:10.328887    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-snhzk" podStartSLOduration=1.440047087 podStartE2EDuration="5.328874437s" totalImagesPullingTime="3.88882735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:10.327951126 +0000 UTC m=+474.228256369" watchObservedRunningTime="2026-07-10 20:29:10.328874437 +0000 UTC m=+474.229179656"
Jul 10 20:29:28.952270 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.952235    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"]
Jul 10 20:29:28.956652 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.956631    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:28.959573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.959552    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 10 20:29:28.959726 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.959687    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 10 20:29:28.959931 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.959912    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 10 20:29:28.960003 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.959933    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zlv5v\""
Jul 10 20:29:28.960003 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.959943    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 10 20:29:28.969444 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.969420    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"]
Jul 10 20:29:28.996855 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.996835    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:28.996947 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.996866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-tmp\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:28.996947 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.996899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:28.996947 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.996922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2bxb\" (UniqueName: \"kubernetes.io/projected/4231385e-84e6-4f1d-8661-37860eb81dea-kube-api-access-f2bxb\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:28.997065 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:28.996952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-manifests\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097327 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097455 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097334    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-tmp\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097542 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097542 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2bxb\" (UniqueName: \"kubernetes.io/projected/4231385e-84e6-4f1d-8661-37860eb81dea-kube-api-access-f2bxb\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097659 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-manifests\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.097932 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.097904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-manifests\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.099458 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.099434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4231385e-84e6-4f1d-8661-37860eb81dea-tmp\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.099748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.099732    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.099793 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.099777    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4231385e-84e6-4f1d-8661-37860eb81dea-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.115610 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.115587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2bxb\" (UniqueName: \"kubernetes.io/projected/4231385e-84e6-4f1d-8661-37860eb81dea-kube-api-access-f2bxb\") pod \"opendatahub-operator-controller-manager-7b6f977c45-5x8q2\" (UID: \"4231385e-84e6-4f1d-8661-37860eb81dea\") " pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.266682 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.266607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:29.393313 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:29.393288    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"]
Jul 10 20:29:29.395917 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:29.395891    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4231385e_84e6_4f1d_8661_37860eb81dea.slice/crio-6035db03bd22d190d3b93d4ae922a4de4a67df4d346697ba8e6706b98684c3d0 WatchSource:0}: Error finding container 6035db03bd22d190d3b93d4ae922a4de4a67df4d346697ba8e6706b98684c3d0: Status 404 returned error can't find the container with id 6035db03bd22d190d3b93d4ae922a4de4a67df4d346697ba8e6706b98684c3d0
Jul 10 20:29:30.387017 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:30.386981    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2" event={"ID":"4231385e-84e6-4f1d-8661-37860eb81dea","Type":"ContainerStarted","Data":"6035db03bd22d190d3b93d4ae922a4de4a67df4d346697ba8e6706b98684c3d0"}
Jul 10 20:29:33.398824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:33.398791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2" event={"ID":"4231385e-84e6-4f1d-8661-37860eb81dea","Type":"ContainerStarted","Data":"9e39d71e09fbf5de49824af355fea7e6eed0c11008d4b665b65974c9bb50ce66"}
Jul 10 20:29:34.403845 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:34.403805    2564 generic.go:356] "Generic (PLEG): container finished" podID="4231385e-84e6-4f1d-8661-37860eb81dea" containerID="9e39d71e09fbf5de49824af355fea7e6eed0c11008d4b665b65974c9bb50ce66" exitCode=0
Jul 10 20:29:34.404310 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:34.403857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2" event={"ID":"4231385e-84e6-4f1d-8661-37860eb81dea","Type":"ContainerDied","Data":"9e39d71e09fbf5de49824af355fea7e6eed0c11008d4b665b65974c9bb50ce66"}
Jul 10 20:29:35.408707 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:35.408672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2" event={"ID":"4231385e-84e6-4f1d-8661-37860eb81dea","Type":"ContainerStarted","Data":"bb02f3056aa815a44b4cae1825b16b334007e56542c7b387f087afe186687805"}
Jul 10 20:29:35.409097 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:35.408817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:35.431102 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:35.431029    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2" podStartSLOduration=2.902951481 podStartE2EDuration="7.431008017s" totalImagesPullingTime="3.501050792s" totalInitContainerRuntime="1.027005744s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:35.428792562 +0000 UTC m=+499.329097781" watchObservedRunningTime="2026-07-10 20:29:35.431008017 +0000 UTC m=+499.331313238"
Jul 10 20:29:46.413938 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:46.413909    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b6f977c45-5x8q2"
Jul 10 20:29:49.312680 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.312649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"]
Jul 10 20:29:49.317293 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.317276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.323566 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.323278    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 10 20:29:49.323566 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.323379    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 10 20:29:49.323716 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.323635    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 10 20:29:49.323716 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.323640    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 10 20:29:49.327311 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.327293    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tls-certs\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.327397 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.327317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tmp\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.327397 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.327349    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h975f\" (UniqueName: \"kubernetes.io/projected/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-kube-api-access-h975f\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.359291 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.359265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"]
Jul 10 20:29:49.427871 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.427845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tls-certs\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.427960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.427878    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tmp\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.427960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.427913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h975f\" (UniqueName: \"kubernetes.io/projected/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-kube-api-access-h975f\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.430063 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.430035    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tmp\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.430182 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.430165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-tls-certs\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.438789 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.438769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h975f\" (UniqueName: \"kubernetes.io/projected/a8f9f0e5-c3d1-43d7-ac43-acf8b297107e-kube-api-access-h975f\") pod \"kube-auth-proxy-b5fcf8fc5-kj45j\" (UID: \"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e\") " pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.561976 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.561952    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9fg6"]
Jul 10 20:29:49.563850 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.563813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.566911 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.566896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fhqm2\""
Jul 10 20:29:49.567581 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.567568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 10 20:29:49.577920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.577899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9fg6"]
Jul 10 20:29:49.626097 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.626072    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"
Jul 10 20:29:49.629839 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.629821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c51fea-0221-4fe7-8438-1a6523469630-cert\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.629920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.629861    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42mkb\" (UniqueName: \"kubernetes.io/projected/81c51fea-0221-4fe7-8438-1a6523469630-kube-api-access-42mkb\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.730879 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.730787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c51fea-0221-4fe7-8438-1a6523469630-cert\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.730879 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.730837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42mkb\" (UniqueName: \"kubernetes.io/projected/81c51fea-0221-4fe7-8438-1a6523469630-kube-api-access-42mkb\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.733179 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.733156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/81c51fea-0221-4fe7-8438-1a6523469630-cert\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.738975 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.738951    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42mkb\" (UniqueName: \"kubernetes.io/projected/81c51fea-0221-4fe7-8438-1a6523469630-kube-api-access-42mkb\") pod \"odh-model-controller-7998c67dc4-n9fg6\" (UID: \"81c51fea-0221-4fe7-8438-1a6523469630\") " pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.752118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.752094    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j"]
Jul 10 20:29:49.755064 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:49.755039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f9f0e5_c3d1_43d7_ac43_acf8b297107e.slice/crio-54e8bc0c1e45db7437f8f30fc5283d0ef38eed7632991177cfce2424b8e3b7ea WatchSource:0}: Error finding container 54e8bc0c1e45db7437f8f30fc5283d0ef38eed7632991177cfce2424b8e3b7ea: Status 404 returned error can't find the container with id 54e8bc0c1e45db7437f8f30fc5283d0ef38eed7632991177cfce2424b8e3b7ea
Jul 10 20:29:49.873226 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.873171    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:49.992961 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:49.992939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n9fg6"]
Jul 10 20:29:49.995114 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:49.995081    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c51fea_0221_4fe7_8438_1a6523469630.slice/crio-c6c0840603cfff824efd8d0bc89e61191ccb2992b5dd2673a2c460a2f902e2e8 WatchSource:0}: Error finding container c6c0840603cfff824efd8d0bc89e61191ccb2992b5dd2673a2c460a2f902e2e8: Status 404 returned error can't find the container with id c6c0840603cfff824efd8d0bc89e61191ccb2992b5dd2673a2c460a2f902e2e8
Jul 10 20:29:50.462516 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.462468    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6" event={"ID":"81c51fea-0221-4fe7-8438-1a6523469630","Type":"ContainerStarted","Data":"c6c0840603cfff824efd8d0bc89e61191ccb2992b5dd2673a2c460a2f902e2e8"}
Jul 10 20:29:50.463897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.463870    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j" event={"ID":"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e","Type":"ContainerStarted","Data":"54e8bc0c1e45db7437f8f30fc5283d0ef38eed7632991177cfce2424b8e3b7ea"}
Jul 10 20:29:50.611281 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.611249    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7j4hq"]
Jul 10 20:29:50.613588 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.613567    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.616722 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.616694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-b2828\""
Jul 10 20:29:50.617009 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.616984    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 10 20:29:50.626423 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.626395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7j4hq"]
Jul 10 20:29:50.635730 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.635708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdslc\" (UniqueName: \"kubernetes.io/projected/b7346fff-038b-4b2d-9366-b09e2e129acf-kube-api-access-jdslc\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.635845 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.635745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.736702 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.736633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdslc\" (UniqueName: \"kubernetes.io/projected/b7346fff-038b-4b2d-9366-b09e2e129acf-kube-api-access-jdslc\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.736702 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.736668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.736845 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:29:50.736798    2564 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 10 20:29:50.736881 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:29:50.736858    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs podName:b7346fff-038b-4b2d-9366-b09e2e129acf nodeName:}" failed. No retries permitted until 2026-07-10 20:29:51.236840916 +0000 UTC m=+515.137146115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs") pod "model-serving-api-54d89bd5fc-7j4hq" (UID: "b7346fff-038b-4b2d-9366-b09e2e129acf") : secret "model-serving-api-tls" not found
Jul 10 20:29:50.751347 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.751319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdslc\" (UniqueName: \"kubernetes.io/projected/b7346fff-038b-4b2d-9366-b09e2e129acf-kube-api-access-jdslc\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:50.836985 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.836957    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:29:50.839930 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.839908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.843119 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.843076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 10 20:29:50.843724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.843537    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9th5z\""
Jul 10 20:29:50.843724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.843557    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 10 20:29:50.843962 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.843786    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 10 20:29:50.845273 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.845250    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 10 20:29:50.853159 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.852994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:29:50.938503 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938661    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938837 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938837 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:50.938952 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:50.938850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2qk\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.040242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.040242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.040443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040265    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.040443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2qk\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.040443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.041229 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.041229 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.040736    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.041723 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.041674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.043139 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.043027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.043242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.043177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.044513 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.044438    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.044603 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.044520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.055220 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.054181    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2qk\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.056554 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.056534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token\") pod \"istiod-openshift-gateway-7574459656-2tm5r\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.152680 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.152648    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:51.242730 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.242695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:51.245848 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.245814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7346fff-038b-4b2d-9366-b09e2e129acf-tls-certs\") pod \"model-serving-api-54d89bd5fc-7j4hq\" (UID: \"b7346fff-038b-4b2d-9366-b09e2e129acf\") " pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:51.307584 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.307471    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:29:51.309902 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:51.309864    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc541292b_d5a6_451c_a4cf_45dcb97e83fd.slice/crio-553eee5cbe4646f005fc154c81dbdf4cedad35eb643ebb4aad526e1feb384f4b WatchSource:0}: Error finding container 553eee5cbe4646f005fc154c81dbdf4cedad35eb643ebb4aad526e1feb384f4b: Status 404 returned error can't find the container with id 553eee5cbe4646f005fc154c81dbdf4cedad35eb643ebb4aad526e1feb384f4b
Jul 10 20:29:51.469302 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.469269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" event={"ID":"c541292b-d5a6-451c-a4cf-45dcb97e83fd","Type":"ContainerStarted","Data":"553eee5cbe4646f005fc154c81dbdf4cedad35eb643ebb4aad526e1feb384f4b"}
Jul 10 20:29:51.526764 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.526660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:51.686459 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:51.686344    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-7j4hq"]
Jul 10 20:29:51.690352 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:29:51.690311    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7346fff_038b_4b2d_9366_b09e2e129acf.slice/crio-0066374a7d6db90b6828c05554747c46cc1b47073b618d288ce5299ce0dcd3e9 WatchSource:0}: Error finding container 0066374a7d6db90b6828c05554747c46cc1b47073b618d288ce5299ce0dcd3e9: Status 404 returned error can't find the container with id 0066374a7d6db90b6828c05554747c46cc1b47073b618d288ce5299ce0dcd3e9
Jul 10 20:29:52.474466 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:52.474428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq" event={"ID":"b7346fff-038b-4b2d-9366-b09e2e129acf","Type":"ContainerStarted","Data":"0066374a7d6db90b6828c05554747c46cc1b47073b618d288ce5299ce0dcd3e9"}
Jul 10 20:29:55.273326 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.273285    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 10 20:29:55.273704 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.273354    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 10 20:29:55.489192 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.489107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq" event={"ID":"b7346fff-038b-4b2d-9366-b09e2e129acf","Type":"ContainerStarted","Data":"97bb7e14ac43c8e230e6c0e8bee0ab686dcd2f9a479edf6996335d229065e912"}
Jul 10 20:29:55.489192 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.489188    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:29:55.490675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.490647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j" event={"ID":"a8f9f0e5-c3d1-43d7-ac43-acf8b297107e","Type":"ContainerStarted","Data":"d751ff293404d06b8149f0ed0707d76b37a946f01e9fe86f1c40afe2a6d1dfca"}
Jul 10 20:29:55.492234 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.492213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" event={"ID":"c541292b-d5a6-451c-a4cf-45dcb97e83fd","Type":"ContainerStarted","Data":"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"}
Jul 10 20:29:55.492343 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.492278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:29:55.493588 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.493566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6" event={"ID":"81c51fea-0221-4fe7-8438-1a6523469630","Type":"ContainerStarted","Data":"f0907b1b0d41eb635ba8608744caeda39a929532347316d6d4f1f831a8d9f789"}
Jul 10 20:29:55.493697 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.493685    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:29:55.524454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.524419    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq" podStartSLOduration=2.006016589 podStartE2EDuration="5.524408686s" totalImagesPullingTime="3.518392097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:55.523116969 +0000 UTC m=+519.423422190" watchObservedRunningTime="2026-07-10 20:29:55.524408686 +0000 UTC m=+519.424713904"
Jul 10 20:29:55.551402 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.551355    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b5fcf8fc5-kj45j" podStartSLOduration=1.10318533 podStartE2EDuration="6.551338363s" totalImagesPullingTime="5.448153033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:55.550002393 +0000 UTC m=+519.450307633" watchObservedRunningTime="2026-07-10 20:29:55.551338363 +0000 UTC m=+519.451643583"
Jul 10 20:29:55.568130 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.568082    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6" podStartSLOduration=2.212206985 podStartE2EDuration="6.568065777s" totalImagesPullingTime="4.355858792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:55.567221783 +0000 UTC m=+519.467527028" watchObservedRunningTime="2026-07-10 20:29:55.568065777 +0000 UTC m=+519.468370998"
Jul 10 20:29:55.596767 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:55.596713    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" podStartSLOduration=1.6367838 podStartE2EDuration="5.59669307s" totalImagesPullingTime="3.95990927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:29:55.595014305 +0000 UTC m=+519.495319524" watchObservedRunningTime="2026-07-10 20:29:55.59669307 +0000 UTC m=+519.496998290"
Jul 10 20:29:56.499183 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:29:56.499153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:30:00.276499 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.276465    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"]
Jul 10 20:30:00.279625 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.279608    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.283050 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.283023    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 10 20:30:00.283136 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.283028    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-2mwt9\""
Jul 10 20:30:00.295954 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.295933    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"]
Jul 10 20:30:00.305294 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.305274    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"]
Jul 10 20:30:00.308242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.308226    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.312033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.312013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 10 20:30:00.312117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.312065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jrh4d\""
Jul 10 20:30:00.317412 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.317389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8hg\" (UniqueName: \"kubernetes.io/projected/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-kube-api-access-xv8hg\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.317518 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.317423    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.330437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.330417    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"]
Jul 10 20:30:00.418030 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.418005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8hg\" (UniqueName: \"kubernetes.io/projected/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-kube-api-access-xv8hg\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.418132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.418037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.418132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.418057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf86\" (UniqueName: \"kubernetes.io/projected/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-kube-api-access-mwf86\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.418132 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.418082    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.420285 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.420256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-cert\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.427673 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.427655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8hg\" (UniqueName: \"kubernetes.io/projected/c0b1fa2a-544b-4182-98a2-0bd8be73b7b3-kube-api-access-xv8hg\") pod \"kserve-controller-manager-d6f4b65b8-ktsqc\" (UID: \"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.518942 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.518916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.519031 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.518944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf86\" (UniqueName: \"kubernetes.io/projected/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-kube-api-access-mwf86\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.519070 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:30:00.519046    2564 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 10 20:30:00.519117 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:30:00.519108    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert podName:20dacbce-46d6-4a6e-97c4-3e7b97b069c9 nodeName:}" failed. No retries permitted until 2026-07-10 20:30:01.019092346 +0000 UTC m=+524.919397542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert") pod "llmisvc-controller-manager-777c6bd7d5-5vhhr" (UID: "20dacbce-46d6-4a6e-97c4-3e7b97b069c9") : secret "llmisvc-webhook-server-cert" not found
Jul 10 20:30:00.542754 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.542697    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf86\" (UniqueName: \"kubernetes.io/projected/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-kube-api-access-mwf86\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:00.588960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.588939    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:00.730798 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:00.730772    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"]
Jul 10 20:30:00.732719 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:30:00.732694    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b1fa2a_544b_4182_98a2_0bd8be73b7b3.slice/crio-f4d768d4daf8389d643c61b1f413e2351dd3c57190687ed47143b7a85fe28b66 WatchSource:0}: Error finding container f4d768d4daf8389d643c61b1f413e2351dd3c57190687ed47143b7a85fe28b66: Status 404 returned error can't find the container with id f4d768d4daf8389d643c61b1f413e2351dd3c57190687ed47143b7a85fe28b66
Jul 10 20:30:01.022625 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.022595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:01.024920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.024902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20dacbce-46d6-4a6e-97c4-3e7b97b069c9-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-5vhhr\" (UID: \"20dacbce-46d6-4a6e-97c4-3e7b97b069c9\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:01.217359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.217328    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:01.366535 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.366512    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"]
Jul 10 20:30:01.368700 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:30:01.368669    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod20dacbce_46d6_4a6e_97c4_3e7b97b069c9.slice/crio-1bb6b3f324f54ebf3f95886513df10e40aef64aa89347cc92a74a11f5610139b WatchSource:0}: Error finding container 1bb6b3f324f54ebf3f95886513df10e40aef64aa89347cc92a74a11f5610139b: Status 404 returned error can't find the container with id 1bb6b3f324f54ebf3f95886513df10e40aef64aa89347cc92a74a11f5610139b
Jul 10 20:30:01.516130 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.516106    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr" event={"ID":"20dacbce-46d6-4a6e-97c4-3e7b97b069c9","Type":"ContainerStarted","Data":"1bb6b3f324f54ebf3f95886513df10e40aef64aa89347cc92a74a11f5610139b"}
Jul 10 20:30:01.517144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:01.517122    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc" event={"ID":"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3","Type":"ContainerStarted","Data":"f4d768d4daf8389d643c61b1f413e2351dd3c57190687ed47143b7a85fe28b66"}
Jul 10 20:30:02.212369 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.212331    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:30:02.212585 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.212557    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" podUID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" containerName="discovery" containerID="cri-o://6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3" gracePeriod=30
Jul 10 20:30:02.457858 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.457836    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:30:02.522748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.522679    2564 generic.go:356] "Generic (PLEG): container finished" podID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" containerID="6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3" exitCode=0
Jul 10 20:30:02.522748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.522731    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"
Jul 10 20:30:02.522894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.522728    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" event={"ID":"c541292b-d5a6-451c-a4cf-45dcb97e83fd","Type":"ContainerDied","Data":"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"}
Jul 10 20:30:02.522894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.522857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r" event={"ID":"c541292b-d5a6-451c-a4cf-45dcb97e83fd","Type":"ContainerDied","Data":"553eee5cbe4646f005fc154c81dbdf4cedad35eb643ebb4aad526e1feb384f4b"}
Jul 10 20:30:02.522894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.522884    2564 scope.go:117] "RemoveContainer" containerID="6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"
Jul 10 20:30:02.531278 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.531264    2564 scope.go:117] "RemoveContainer" containerID="6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"
Jul 10 20:30:02.531632 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:30:02.531605    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3\": container with ID starting with 6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3 not found: ID does not exist" containerID="6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"
Jul 10 20:30:02.531707 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.531632    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3"} err="failed to get container status \"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3\": rpc error: code = NotFound desc = could not find container \"6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3\": container with ID starting with 6adfb276353d0a76552234adde0345bbb377f6e0d95f3cb8d223fead38d05fa3 not found: ID does not exist"
Jul 10 20:30:02.534921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.534900    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535026 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.534936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pv2qk\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535026 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.534959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535026 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.534998    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535174 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.535030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535174 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.535056    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535174 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.535125    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs\") pod \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\" (UID: \"c541292b-d5a6-451c-a4cf-45dcb97e83fd\") "
Jul 10 20:30:02.535396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.535369    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:30:02.537704 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.537678    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:30:02.537904 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.537874    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:30:02.538123 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.538091    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:30:02.538385 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.538363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs" (OuterVolumeSpecName: "local-certs") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:30:02.538458 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.538402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts" (OuterVolumeSpecName: "cacerts") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:30:02.538512 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.538454    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk" (OuterVolumeSpecName: "kube-api-access-pv2qk") pod "c541292b-d5a6-451c-a4cf-45dcb97e83fd" (UID: "c541292b-d5a6-451c-a4cf-45dcb97e83fd"). InnerVolumeSpecName "kube-api-access-pv2qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:30:02.635935 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635903    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-kubeconfig\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.635935 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635927    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-dns-cert\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.635935 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635937    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c541292b-d5a6-451c-a4cf-45dcb97e83fd-local-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.635935 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635945    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-csr-ca-configmap\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.636264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635954    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2qk\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-kube-api-access-pv2qk\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.636264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635963    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c541292b-d5a6-451c-a4cf-45dcb97e83fd-istio-token\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.636264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.635972    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c541292b-d5a6-451c-a4cf-45dcb97e83fd-cacerts\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:02.847312 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.847284    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:30:02.852356 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:02.852330    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-2tm5r"]
Jul 10 20:30:04.535654 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:04.535617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc" event={"ID":"c0b1fa2a-544b-4182-98a2-0bd8be73b7b3","Type":"ContainerStarted","Data":"1004748a6562043dc54f3d4aa948adce180e0c8994e056c227babab9dde304cd"}
Jul 10 20:30:04.536079 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:04.535733    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:04.554629 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:04.554576    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc" podStartSLOduration=1.281997532 podStartE2EDuration="4.554559869s" totalImagesPullingTime="3.272562337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:30:04.552748244 +0000 UTC m=+528.453053485" watchObservedRunningTime="2026-07-10 20:30:04.554559869 +0000 UTC m=+528.454865087"
Jul 10 20:30:04.784110 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:04.784073    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" path="/var/lib/kubelet/pods/c541292b-d5a6-451c-a4cf-45dcb97e83fd/volumes"
Jul 10 20:30:05.540876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:05.540839    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr" event={"ID":"20dacbce-46d6-4a6e-97c4-3e7b97b069c9","Type":"ContainerStarted","Data":"e31e87310eecb33107d353f1cdf1d0cb4121893b48dd2aa3ff953fe6c3ba3e81"}
Jul 10 20:30:05.557472 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:05.557425    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr" podStartSLOduration=1.951090258 podStartE2EDuration="5.557410983s" totalImagesPullingTime="3.606320725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:30:05.557340906 +0000 UTC m=+529.457646136" watchObservedRunningTime="2026-07-10 20:30:05.557410983 +0000 UTC m=+529.457716203"
Jul 10 20:30:06.499549 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:06.499522    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n9fg6"
Jul 10 20:30:06.502645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:06.502625    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-7j4hq"
Jul 10 20:30:06.545131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:06.545106    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:35.546239 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:35.546208    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ktsqc"
Jul 10 20:30:37.551969 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:37.551940    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-5vhhr"
Jul 10 20:30:50.222958 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.222925    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:50.223391 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.223187    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" containerName="discovery"
Jul 10 20:30:50.223391 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.223199    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" containerName="discovery"
Jul 10 20:30:50.223391 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.223265    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c541292b-d5a6-451c-a4cf-45dcb97e83fd" containerName="discovery"
Jul 10 20:30:50.230317 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.230290    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:50.232873 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.232847    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 10 20:30:50.234082 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.234059    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 10 20:30:50.234398 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.234085    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dbf28\""
Jul 10 20:30:50.236156 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.236112    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:50.358780 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.358755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc5cb\" (UniqueName: \"kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb\") pod \"kuadrant-operator-catalog-dnnc2\" (UID: \"67523118-52b3-42b2-a9e8-836dec6044d6\") " pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:50.459373 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.459353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc5cb\" (UniqueName: \"kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb\") pod \"kuadrant-operator-catalog-dnnc2\" (UID: \"67523118-52b3-42b2-a9e8-836dec6044d6\") " pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:50.468207 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.468185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc5cb\" (UniqueName: \"kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb\") pod \"kuadrant-operator-catalog-dnnc2\" (UID: \"67523118-52b3-42b2-a9e8-836dec6044d6\") " pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:50.540938 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.540885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:50.593298 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.593268    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:50.666333 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.666310    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:50.668748 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:30:50.668720    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67523118_52b3_42b2_a9e8_836dec6044d6.slice/crio-e107cad531d63a7bc6a7f6195e1368cdbae14cb60e19a5023473ff5f2cbacb2d WatchSource:0}: Error finding container e107cad531d63a7bc6a7f6195e1368cdbae14cb60e19a5023473ff5f2cbacb2d: Status 404 returned error can't find the container with id e107cad531d63a7bc6a7f6195e1368cdbae14cb60e19a5023473ff5f2cbacb2d
Jul 10 20:30:50.710460 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.710427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" event={"ID":"67523118-52b3-42b2-a9e8-836dec6044d6","Type":"ContainerStarted","Data":"e107cad531d63a7bc6a7f6195e1368cdbae14cb60e19a5023473ff5f2cbacb2d"}
Jul 10 20:30:50.801527 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.801461    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvst7"]
Jul 10 20:30:50.805671 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.805657    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:30:50.812003 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.811971    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvst7"]
Jul 10 20:30:50.962366 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:50.962336    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xthl\" (UniqueName: \"kubernetes.io/projected/8455dd04-f0e8-4379-98b4-c09e31b37677-kube-api-access-8xthl\") pod \"kuadrant-operator-catalog-jvst7\" (UID: \"8455dd04-f0e8-4379-98b4-c09e31b37677\") " pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:30:51.062996 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:51.062976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xthl\" (UniqueName: \"kubernetes.io/projected/8455dd04-f0e8-4379-98b4-c09e31b37677-kube-api-access-8xthl\") pod \"kuadrant-operator-catalog-jvst7\" (UID: \"8455dd04-f0e8-4379-98b4-c09e31b37677\") " pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:30:51.070621 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:51.070602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xthl\" (UniqueName: \"kubernetes.io/projected/8455dd04-f0e8-4379-98b4-c09e31b37677-kube-api-access-8xthl\") pod \"kuadrant-operator-catalog-jvst7\" (UID: \"8455dd04-f0e8-4379-98b4-c09e31b37677\") " pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:30:51.115622 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:51.115599    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:30:51.267825 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:51.267654    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvst7"]
Jul 10 20:30:51.289439 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:30:51.289415    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8455dd04_f0e8_4379_98b4_c09e31b37677.slice/crio-29fd322ddb8a19ec5d0ea411f1f0b1b51560923e4e45ef1216b7b58e301d8220 WatchSource:0}: Error finding container 29fd322ddb8a19ec5d0ea411f1f0b1b51560923e4e45ef1216b7b58e301d8220: Status 404 returned error can't find the container with id 29fd322ddb8a19ec5d0ea411f1f0b1b51560923e4e45ef1216b7b58e301d8220
Jul 10 20:30:51.716806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:51.716746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvst7" event={"ID":"8455dd04-f0e8-4379-98b4-c09e31b37677","Type":"ContainerStarted","Data":"29fd322ddb8a19ec5d0ea411f1f0b1b51560923e4e45ef1216b7b58e301d8220"}
Jul 10 20:30:52.722078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.722040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvst7" event={"ID":"8455dd04-f0e8-4379-98b4-c09e31b37677","Type":"ContainerStarted","Data":"033bb68b41045114192cbe41f25916e5cc26b489c70424791f5fa3c172c79637"}
Jul 10 20:30:52.723589 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.723552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" event={"ID":"67523118-52b3-42b2-a9e8-836dec6044d6","Type":"ContainerStarted","Data":"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"}
Jul 10 20:30:52.723735 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.723642    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" podUID="67523118-52b3-42b2-a9e8-836dec6044d6" containerName="registry-server" containerID="cri-o://601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b" gracePeriod=2
Jul 10 20:30:52.737884 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.737839    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jvst7" podStartSLOduration=1.477764989 podStartE2EDuration="2.737827645s" totalImagesPullingTime="1.260062656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:30:52.73696171 +0000 UTC m=+576.637266930" watchObservedRunningTime="2026-07-10 20:30:52.737827645 +0000 UTC m=+576.638132864"
Jul 10 20:30:52.752177 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.752141    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" podStartSLOduration=0.872034628 podStartE2EDuration="2.752128847s" totalImagesPullingTime="1.880094219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:30:52.751145962 +0000 UTC m=+576.651451187" watchObservedRunningTime="2026-07-10 20:30:52.752128847 +0000 UTC m=+576.652434065"
Jul 10 20:30:52.957392 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:52.957369    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:53.086597 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.086559    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc5cb\" (UniqueName: \"kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb\") pod \"67523118-52b3-42b2-a9e8-836dec6044d6\" (UID: \"67523118-52b3-42b2-a9e8-836dec6044d6\") "
Jul 10 20:30:53.088585 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.088561    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb" (OuterVolumeSpecName: "kube-api-access-fc5cb") pod "67523118-52b3-42b2-a9e8-836dec6044d6" (UID: "67523118-52b3-42b2-a9e8-836dec6044d6"). InnerVolumeSpecName "kube-api-access-fc5cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:30:53.187021 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.186961    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc5cb\" (UniqueName: \"kubernetes.io/projected/67523118-52b3-42b2-a9e8-836dec6044d6-kube-api-access-fc5cb\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:30:53.728770 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.728740    2564 generic.go:356] "Generic (PLEG): container finished" podID="67523118-52b3-42b2-a9e8-836dec6044d6" containerID="601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b" exitCode=0
Jul 10 20:30:53.729131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.728778    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2"
Jul 10 20:30:53.729131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.728830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" event={"ID":"67523118-52b3-42b2-a9e8-836dec6044d6","Type":"ContainerDied","Data":"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"}
Jul 10 20:30:53.729131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.728876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dnnc2" event={"ID":"67523118-52b3-42b2-a9e8-836dec6044d6","Type":"ContainerDied","Data":"e107cad531d63a7bc6a7f6195e1368cdbae14cb60e19a5023473ff5f2cbacb2d"}
Jul 10 20:30:53.729131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.728901    2564 scope.go:117] "RemoveContainer" containerID="601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"
Jul 10 20:30:53.738143 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.738129    2564 scope.go:117] "RemoveContainer" containerID="601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"
Jul 10 20:30:53.738394 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:30:53.738373    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b\": container with ID starting with 601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b not found: ID does not exist" containerID="601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"
Jul 10 20:30:53.738451 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.738405    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b"} err="failed to get container status \"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b\": rpc error: code = NotFound desc = could not find container \"601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b\": container with ID starting with 601c194bd489f6839885d89f6696ba0a5c6e7afd6924799cabe6c36f31f4536b not found: ID does not exist"
Jul 10 20:30:53.749095 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.749073    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:53.753237 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:53.753218    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dnnc2"]
Jul 10 20:30:54.783794 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:30:54.783763    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67523118-52b3-42b2-a9e8-836dec6044d6" path="/var/lib/kubelet/pods/67523118-52b3-42b2-a9e8-836dec6044d6/volumes"
Jul 10 20:31:01.116784 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:01.116749    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:31:01.117256 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:01.116819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:31:01.138843 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:01.138815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:31:01.788013 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:01.787986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jvst7"
Jul 10 20:31:16.629252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:16.629230    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:31:16.630224 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:16.630206    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:31:24.597249 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.597217    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"]
Jul 10 20:31:24.603117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.597529    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67523118-52b3-42b2-a9e8-836dec6044d6" containerName="registry-server"
Jul 10 20:31:24.603117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.597543    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67523118-52b3-42b2-a9e8-836dec6044d6" containerName="registry-server"
Jul 10 20:31:24.603117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.597596    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67523118-52b3-42b2-a9e8-836dec6044d6" containerName="registry-server"
Jul 10 20:31:24.603630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.603615    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.606607 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.606588    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gbr7t\""
Jul 10 20:31:24.618142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.618123    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"]
Jul 10 20:31:24.686983 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.686964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.687083 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.686993    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b24c\" (UniqueName: \"kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.787554 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.787524    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.787699 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.787561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b24c\" (UniqueName: \"kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.787832 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.787812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.798127 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.798103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b24c\" (UniqueName: \"kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:24.912965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:24.912912    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:25.036205 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:25.036148    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"]
Jul 10 20:31:25.039142 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:25.039112    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97833ffd_2b65_4bf5_9ccd_b2a2eba5dc4a.slice/crio-b868d6eff7ccbaa35ab0be08cef1859878e0e1e098d0d71097433d76a7ac35aa WatchSource:0}: Error finding container b868d6eff7ccbaa35ab0be08cef1859878e0e1e098d0d71097433d76a7ac35aa: Status 404 returned error can't find the container with id b868d6eff7ccbaa35ab0be08cef1859878e0e1e098d0d71097433d76a7ac35aa
Jul 10 20:31:25.861560 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:25.861524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4" event={"ID":"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a","Type":"ContainerStarted","Data":"b868d6eff7ccbaa35ab0be08cef1859878e0e1e098d0d71097433d76a7ac35aa"}
Jul 10 20:31:26.541961 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.541931    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"]
Jul 10 20:31:26.545118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.545101    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:26.548503 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.548464    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 10 20:31:26.548960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.548947    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-42vkx\""
Jul 10 20:31:26.568130 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.568110    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"]
Jul 10 20:31:26.599031 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.599006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcn8s\" (UniqueName: \"kubernetes.io/projected/f2a308e0-f01f-47a8-9015-783d08471b7f-kube-api-access-tcn8s\") pod \"dns-operator-controller-manager-864dd78c4c-s77sx\" (UID: \"f2a308e0-f01f-47a8-9015-783d08471b7f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:26.699233 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.699210    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcn8s\" (UniqueName: \"kubernetes.io/projected/f2a308e0-f01f-47a8-9015-783d08471b7f-kube-api-access-tcn8s\") pod \"dns-operator-controller-manager-864dd78c4c-s77sx\" (UID: \"f2a308e0-f01f-47a8-9015-783d08471b7f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:26.717451 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.717425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcn8s\" (UniqueName: \"kubernetes.io/projected/f2a308e0-f01f-47a8-9015-783d08471b7f-kube-api-access-tcn8s\") pod \"dns-operator-controller-manager-864dd78c4c-s77sx\" (UID: \"f2a308e0-f01f-47a8-9015-783d08471b7f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:26.854992 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.854961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:26.980886 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:26.980856    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"]
Jul 10 20:31:26.985646 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:26.985609    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a308e0_f01f_47a8_9015_783d08471b7f.slice/crio-14d3b08d04c8e9191b0085d9e939476d731a81554fc7bd5f0875b155c060a45d WatchSource:0}: Error finding container 14d3b08d04c8e9191b0085d9e939476d731a81554fc7bd5f0875b155c060a45d: Status 404 returned error can't find the container with id 14d3b08d04c8e9191b0085d9e939476d731a81554fc7bd5f0875b155c060a45d
Jul 10 20:31:27.873572 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:27.873525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx" event={"ID":"f2a308e0-f01f-47a8-9015-783d08471b7f","Type":"ContainerStarted","Data":"14d3b08d04c8e9191b0085d9e939476d731a81554fc7bd5f0875b155c060a45d"}
Jul 10 20:31:29.846468 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.846441    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"]
Jul 10 20:31:29.849914 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.849890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:29.852796 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.852771    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-xkfvw\""
Jul 10 20:31:29.860396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.860371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"]
Jul 10 20:31:29.884856 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.884829    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx" event={"ID":"f2a308e0-f01f-47a8-9015-783d08471b7f","Type":"ContainerStarted","Data":"8ba30144b8251f88b4d6faba32f421cef5ee0eacbb7027c482694d78cac8f8e5"}
Jul 10 20:31:29.884967 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.884955    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:29.904235 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.904172    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx" podStartSLOduration=1.890120915 podStartE2EDuration="3.904154656s" totalImagesPullingTime="2.014033741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:29.9013382 +0000 UTC m=+613.801643418" watchObservedRunningTime="2026-07-10 20:31:29.904154656 +0000 UTC m=+613.804459877"
Jul 10 20:31:29.923024 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:29.922995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf4ls\" (UniqueName: \"kubernetes.io/projected/4f83aa55-b54c-4057-9218-6f897706ff1d-kube-api-access-cf4ls\") pod \"authorino-operator-75cc8d5cb7-h9lj5\" (UID: \"4f83aa55-b54c-4057-9218-6f897706ff1d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:30.023879 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:30.023842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf4ls\" (UniqueName: \"kubernetes.io/projected/4f83aa55-b54c-4057-9218-6f897706ff1d-kube-api-access-cf4ls\") pod \"authorino-operator-75cc8d5cb7-h9lj5\" (UID: \"4f83aa55-b54c-4057-9218-6f897706ff1d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:30.036374 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:30.036341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf4ls\" (UniqueName: \"kubernetes.io/projected/4f83aa55-b54c-4057-9218-6f897706ff1d-kube-api-access-cf4ls\") pod \"authorino-operator-75cc8d5cb7-h9lj5\" (UID: \"4f83aa55-b54c-4057-9218-6f897706ff1d\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:30.161100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:30.161028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:31.078924 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:31.078902    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"]
Jul 10 20:31:31.080760 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:31.080736    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f83aa55_b54c_4057_9218_6f897706ff1d.slice/crio-ed24a87fbc8a855dbc39711bb6359a2027de3ad8aa6d7614d096e32dbd6c9884 WatchSource:0}: Error finding container ed24a87fbc8a855dbc39711bb6359a2027de3ad8aa6d7614d096e32dbd6c9884: Status 404 returned error can't find the container with id ed24a87fbc8a855dbc39711bb6359a2027de3ad8aa6d7614d096e32dbd6c9884
Jul 10 20:31:31.900042 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:31.899638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4" event={"ID":"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a","Type":"ContainerStarted","Data":"5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7"}
Jul 10 20:31:31.900042 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:31.899738    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:31.902259 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:31.902220    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5" event={"ID":"4f83aa55-b54c-4057-9218-6f897706ff1d","Type":"ContainerStarted","Data":"ed24a87fbc8a855dbc39711bb6359a2027de3ad8aa6d7614d096e32dbd6c9884"}
Jul 10 20:31:31.919724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:31.919672    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4" podStartSLOduration=1.9461918470000001 podStartE2EDuration="7.91965806s" totalImagesPullingTime="5.973466213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:31.917843987 +0000 UTC m=+615.818149239" watchObservedRunningTime="2026-07-10 20:31:31.91965806 +0000 UTC m=+615.819963280"
Jul 10 20:31:32.570100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.570071    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"]
Jul 10 20:31:32.574550 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.574530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:32.577372 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.577350    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kkpbn\""
Jul 10 20:31:32.585386 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.585343    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"]
Jul 10 20:31:32.643774 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.643743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdrv\" (UniqueName: \"kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv\") pod \"limitador-operator-controller-manager-859d7b78bd-d68br\" (UID: \"32fd991e-0b03-4f69-99ac-66565162606b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:32.744622 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.744585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdrv\" (UniqueName: \"kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv\") pod \"limitador-operator-controller-manager-859d7b78bd-d68br\" (UID: \"32fd991e-0b03-4f69-99ac-66565162606b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:32.754044 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.754009    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdrv\" (UniqueName: \"kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv\") pod \"limitador-operator-controller-manager-859d7b78bd-d68br\" (UID: \"32fd991e-0b03-4f69-99ac-66565162606b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:32.886752 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:32.886674    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:33.227026 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:33.227004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"]
Jul 10 20:31:33.228895 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:33.228872    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd991e_0b03_4f69_99ac_66565162606b.slice/crio-aa7dd739a918d43b22c7a1be5275cc931c1421487e778139e30ed93bd1563aa5 WatchSource:0}: Error finding container aa7dd739a918d43b22c7a1be5275cc931c1421487e778139e30ed93bd1563aa5: Status 404 returned error can't find the container with id aa7dd739a918d43b22c7a1be5275cc931c1421487e778139e30ed93bd1563aa5
Jul 10 20:31:33.913441 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:33.913404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5" event={"ID":"4f83aa55-b54c-4057-9218-6f897706ff1d","Type":"ContainerStarted","Data":"c34c2c5e3f1c1609a67054f31ced741b279a6bf33e7b105f8250560b0fe55397"}
Jul 10 20:31:33.913859 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:33.913701    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:33.915093 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:33.915068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br" event={"ID":"32fd991e-0b03-4f69-99ac-66565162606b","Type":"ContainerStarted","Data":"aa7dd739a918d43b22c7a1be5275cc931c1421487e778139e30ed93bd1563aa5"}
Jul 10 20:31:33.942207 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:33.942159    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5" podStartSLOduration=2.862561326 podStartE2EDuration="4.942141181s" totalImagesPullingTime="2.079579855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:33.941371662 +0000 UTC m=+617.841676918" watchObservedRunningTime="2026-07-10 20:31:33.942141181 +0000 UTC m=+617.842446401"
Jul 10 20:31:34.182608 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.182526    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"]
Jul 10 20:31:34.186183 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.186160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.189167 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.189141    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nzk8q\""
Jul 10 20:31:34.189302 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.189178    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 10 20:31:34.189302 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.189190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 10 20:31:34.194059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.194036    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"]
Jul 10 20:31:34.256384 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.256361    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/457920d3-7eac-4752-ab7b-f9d389f983b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.256479 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.256398    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhg7\" (UniqueName: \"kubernetes.io/projected/457920d3-7eac-4752-ab7b-f9d389f983b2-kube-api-access-mvhg7\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.256479 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.256435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/457920d3-7eac-4752-ab7b-f9d389f983b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.357436 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.357406    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/457920d3-7eac-4752-ab7b-f9d389f983b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.357583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.357441    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhg7\" (UniqueName: \"kubernetes.io/projected/457920d3-7eac-4752-ab7b-f9d389f983b2-kube-api-access-mvhg7\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.357583 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.357464    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/457920d3-7eac-4752-ab7b-f9d389f983b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.358088 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.358069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/457920d3-7eac-4752-ab7b-f9d389f983b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.359698 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.359680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/457920d3-7eac-4752-ab7b-f9d389f983b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.366424 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.366385    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhg7\" (UniqueName: \"kubernetes.io/projected/457920d3-7eac-4752-ab7b-f9d389f983b2-kube-api-access-mvhg7\") pod \"kuadrant-console-plugin-66b5774985-kfw5w\" (UID: \"457920d3-7eac-4752-ab7b-f9d389f983b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.498199 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.498142    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"
Jul 10 20:31:34.627291 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.627265    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w"]
Jul 10 20:31:34.629616 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:34.629588    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod457920d3_7eac_4752_ab7b_f9d389f983b2.slice/crio-38037a62936ce68df5ba63d353861efd132bcf29dda23cde55ca67a7387eae0d WatchSource:0}: Error finding container 38037a62936ce68df5ba63d353861efd132bcf29dda23cde55ca67a7387eae0d: Status 404 returned error can't find the container with id 38037a62936ce68df5ba63d353861efd132bcf29dda23cde55ca67a7387eae0d
Jul 10 20:31:34.919845 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:34.919806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w" event={"ID":"457920d3-7eac-4752-ab7b-f9d389f983b2","Type":"ContainerStarted","Data":"38037a62936ce68df5ba63d353861efd132bcf29dda23cde55ca67a7387eae0d"}
Jul 10 20:31:35.927870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:35.927830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br" event={"ID":"32fd991e-0b03-4f69-99ac-66565162606b","Type":"ContainerStarted","Data":"1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a"}
Jul 10 20:31:35.928303 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:35.927945    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:35.946419 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:35.946358    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br" podStartSLOduration=1.8251783499999998 podStartE2EDuration="3.946338349s" totalImagesPullingTime="2.121159999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:35.944665877 +0000 UTC m=+619.844971096" watchObservedRunningTime="2026-07-10 20:31:35.946338349 +0000 UTC m=+619.846643569"
Jul 10 20:31:40.892233 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:40.892163    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-s77sx"
Jul 10 20:31:42.909870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:42.909837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:44.585611 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.585575    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"]
Jul 10 20:31:44.586015 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.585786    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" containerName="manager" containerID="cri-o://5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7" gracePeriod=2
Jul 10 20:31:44.593376 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.593336    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\": pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:44.594465 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.594406    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"]
Jul 10 20:31:44.603750 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.603716    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"]
Jul 10 20:31:44.604253 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.603914    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br" podUID="32fd991e-0b03-4f69-99ac-66565162606b" containerName="manager" containerID="cri-o://1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a" gracePeriod=2
Jul 10 20:31:44.606171 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.606150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:44.607596 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.607458    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:44.613018 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.612874    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:31:44.613275 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.613248    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" containerName="manager"
Jul 10 20:31:44.613275 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.613267    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" containerName="manager"
Jul 10 20:31:44.613446 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.613347    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" containerName="manager"
Jul 10 20:31:44.617031 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.616749    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.618687 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.618640    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"]
Jul 10 20:31:44.623448 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.623378    2564 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"32fd991e-0b03-4f69-99ac-66565162606b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-d68br\": pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:44.638054 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.637969    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:31:44.639454 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.639422    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"]
Jul 10 20:31:44.639755 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.639738    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32fd991e-0b03-4f69-99ac-66565162606b" containerName="manager"
Jul 10 20:31:44.639835 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.639758    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="32fd991e-0b03-4f69-99ac-66565162606b" containerName="manager"
Jul 10 20:31:44.639876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.639849    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="32fd991e-0b03-4f69-99ac-66565162606b" containerName="manager"
Jul 10 20:31:44.643856 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.643825    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:44.653154 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.653099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"]
Jul 10 20:31:44.656541 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.656480    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:44.658646 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.658614    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:44.661054 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.661027    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:44.680223 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:44.680189    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:44.735084 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.735057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7dz\" (UniqueName: \"kubernetes.io/projected/902aac0c-29f6-4f7f-a53c-fffc6a69ce25-kube-api-access-wb7dz\") pod \"limitador-operator-controller-manager-859d7b78bd-th8zc\" (UID: \"902aac0c-29f6-4f7f-a53c-fffc6a69ce25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:44.735194 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.735130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqwkn\" (UniqueName: \"kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.735251 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.735211    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.836294 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.836222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqwkn\" (UniqueName: \"kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.836294 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.836289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.836528 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.836343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7dz\" (UniqueName: \"kubernetes.io/projected/902aac0c-29f6-4f7f-a53c-fffc6a69ce25-kube-api-access-wb7dz\") pod \"limitador-operator-controller-manager-859d7b78bd-th8zc\" (UID: \"902aac0c-29f6-4f7f-a53c-fffc6a69ce25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:44.836734 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.836713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.852452 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.852422    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7dz\" (UniqueName: \"kubernetes.io/projected/902aac0c-29f6-4f7f-a53c-fffc6a69ce25-kube-api-access-wb7dz\") pod \"limitador-operator-controller-manager-859d7b78bd-th8zc\" (UID: \"902aac0c-29f6-4f7f-a53c-fffc6a69ce25\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:44.852722 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.852704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqwkn\" (UniqueName: \"kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tvxf8\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:44.922635 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:44.922612    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-h9lj5"
Jul 10 20:31:45.004412 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:45.004380    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:45.012014 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:45.011994    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:57.546126 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.546102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:57.548683 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:57.548651    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:57.550072 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.550053    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:31:57.552617 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:57.552585    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f043733_4943_476f_bfe4_d3dab8f2ff8b.slice/crio-007cc5a1f525964a000eac0d4fcf87dc843c4ef9d2cc2ca11eecee710d58e24a WatchSource:0}: Error finding container 007cc5a1f525964a000eac0d4fcf87dc843c4ef9d2cc2ca11eecee710d58e24a: Status 404 returned error can't find the container with id 007cc5a1f525964a000eac0d4fcf87dc843c4ef9d2cc2ca11eecee710d58e24a
Jul 10 20:31:57.559282 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.559260    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:57.561649 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.561628    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"]
Jul 10 20:31:57.561717 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:57.561626    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:57.563801 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:57.563778    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:57.565636 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:31:57.565612    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902aac0c_29f6_4f7f_a53c_fffc6a69ce25.slice/crio-e7f1068b7a642a496a7e65b80eb67dd7f55e15e3d0bfbc41cad72f936372aabb WatchSource:0}: Error finding container e7f1068b7a642a496a7e65b80eb67dd7f55e15e3d0bfbc41cad72f936372aabb: Status 404 returned error can't find the container with id e7f1068b7a642a496a7e65b80eb67dd7f55e15e3d0bfbc41cad72f936372aabb
Jul 10 20:31:57.643388 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.643346    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume\") pod \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") "
Jul 10 20:31:57.643482 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.643403    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b24c\" (UniqueName: \"kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c\") pod \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\" (UID: \"97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a\") "
Jul 10 20:31:57.643482 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.643432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfdrv\" (UniqueName: \"kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv\") pod \"32fd991e-0b03-4f69-99ac-66565162606b\" (UID: \"32fd991e-0b03-4f69-99ac-66565162606b\") "
Jul 10 20:31:57.643888 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.643855    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" (UID: "97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:31:57.645158 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.645137    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c" (OuterVolumeSpecName: "kube-api-access-6b24c") pod "97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" (UID: "97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a"). InnerVolumeSpecName "kube-api-access-6b24c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:31:57.645235 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.645214    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv" (OuterVolumeSpecName: "kube-api-access-vfdrv") pod "32fd991e-0b03-4f69-99ac-66565162606b" (UID: "32fd991e-0b03-4f69-99ac-66565162606b"). InnerVolumeSpecName "kube-api-access-vfdrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:31:57.744727 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.744701    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-extensions-socket-volume\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:31:57.744727 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.744723    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b24c\" (UniqueName: \"kubernetes.io/projected/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a-kube-api-access-6b24c\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:31:57.744860 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:57.744733    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdrv\" (UniqueName: \"kubernetes.io/projected/32fd991e-0b03-4f69-99ac-66565162606b-kube-api-access-vfdrv\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:31:58.047633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.047562    2564 generic.go:356] "Generic (PLEG): container finished" podID="32fd991e-0b03-4f69-99ac-66565162606b" containerID="1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a" exitCode=0
Jul 10 20:31:58.047633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.047605    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:58.047819 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.047667    2564 scope.go:117] "RemoveContainer" containerID="1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a"
Jul 10 20:31:58.049088 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.049064    2564 generic.go:356] "Generic (PLEG): container finished" podID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" containerID="5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7" exitCode=0
Jul 10 20:31:58.049207 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.049110    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:58.050361 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.050331    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:58.050878 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.050851    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" event={"ID":"5f043733-4943-476f-bfe4-d3dab8f2ff8b","Type":"ContainerStarted","Data":"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"}
Jul 10 20:31:58.050990 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.050915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" event={"ID":"5f043733-4943-476f-bfe4-d3dab8f2ff8b","Type":"ContainerStarted","Data":"007cc5a1f525964a000eac0d4fcf87dc843c4ef9d2cc2ca11eecee710d58e24a"}
Jul 10 20:31:58.050990 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.050946    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:31:58.052591 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.052568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc" event={"ID":"902aac0c-29f6-4f7f-a53c-fffc6a69ce25","Type":"ContainerStarted","Data":"a3b2f87ec52b4dcaba4d9d1d32de5edc0ed67ffa4e5c4479c72ffc4abdf0d204"}
Jul 10 20:31:58.052692 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.052611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc" event={"ID":"902aac0c-29f6-4f7f-a53c-fffc6a69ce25","Type":"ContainerStarted","Data":"e7f1068b7a642a496a7e65b80eb67dd7f55e15e3d0bfbc41cad72f936372aabb"}
Jul 10 20:31:58.052755 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.052684    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:58.052755 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.052724    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:31:58.054288 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.054265    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w" event={"ID":"457920d3-7eac-4752-ab7b-f9d389f983b2","Type":"ContainerStarted","Data":"fb2ff5ef38f1928792850c614a4fa7015686016f0af1d68e1af618e5f3988c34"}
Jul 10 20:31:58.054915 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.054894    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:58.057880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.057016    2564 scope.go:117] "RemoveContainer" containerID="1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a"
Jul 10 20:31:58.057880 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.057549    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a\": container with ID starting with 1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a not found: ID does not exist" containerID="1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a"
Jul 10 20:31:58.057880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.057581    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a"} err="failed to get container status \"1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a\": rpc error: code = NotFound desc = could not find container \"1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a\": container with ID starting with 1a1f35016212b993d93ccf5ff5bd38ec4040646a6d3d6ba446d8ed2be195f14a not found: ID does not exist"
Jul 10 20:31:58.057880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.057604    2564 scope.go:117] "RemoveContainer" containerID="5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7"
Jul 10 20:31:58.066371 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.066352    2564 scope.go:117] "RemoveContainer" containerID="5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7"
Jul 10 20:31:58.066633 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.066616    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7\": container with ID starting with 5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7 not found: ID does not exist" containerID="5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7"
Jul 10 20:31:58.066688 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.066638    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7"} err="failed to get container status \"5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7\": rpc error: code = NotFound desc = could not find container \"5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7\": container with ID starting with 5b036e6707bd6beae51346ffdce8bf6dc789c09c835552be35347f4ac41eeaa7 not found: ID does not exist"
Jul 10 20:31:58.075427 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.075390    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" podStartSLOduration=14.075378222 podStartE2EDuration="14.075378222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:58.073558693 +0000 UTC m=+641.973863945" watchObservedRunningTime="2026-07-10 20:31:58.075378222 +0000 UTC m=+641.975683441"
Jul 10 20:31:58.076066 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.076045    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:58.093962 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.093924    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc" podStartSLOduration=14.093912778 podStartE2EDuration="14.093912778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:58.09278568 +0000 UTC m=+641.993090920" watchObservedRunningTime="2026-07-10 20:31:58.093912778 +0000 UTC m=+641.994217997"
Jul 10 20:31:58.095002 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.094983    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-d68br\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="32fd991e-0b03-4f69-99ac-66565162606b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d68br"
Jul 10 20:31:58.113287 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.113250    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-kfw5w" podStartSLOduration=1.2691211070000001 podStartE2EDuration="24.113238207s" totalImagesPullingTime="22.8441171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:31:58.112785667 +0000 UTC m=+642.013090886" watchObservedRunningTime="2026-07-10 20:31:58.113238207 +0000 UTC m=+642.013543425"
Jul 10 20:31:58.114974 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:31:58.114954    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2nvd4\" is forbidden: User \"system:node:ip-10-0-141-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-240.ec2.internal' and this object" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2nvd4"
Jul 10 20:31:58.784819 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.784745    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32fd991e-0b03-4f69-99ac-66565162606b" path="/var/lib/kubelet/pods/32fd991e-0b03-4f69-99ac-66565162606b/volumes"
Jul 10 20:31:58.785161 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:31:58.785055    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a" path="/var/lib/kubelet/pods/97833ffd-2b65-4bf5-9ccd-b2a2eba5dc4a/volumes"
Jul 10 20:32:03.397693 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.397660    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:32:03.398272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.397830    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" podUID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" containerName="manager" containerID="cri-o://00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae" gracePeriod=10
Jul 10 20:32:03.399747 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.399723    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:32:03.666767 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.666711    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:32:03.784584 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.784554    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqwkn\" (UniqueName: \"kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn\") pod \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") "
Jul 10 20:32:03.784751 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.784609    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume\") pod \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\" (UID: \"5f043733-4943-476f-bfe4-d3dab8f2ff8b\") "
Jul 10 20:32:03.784989 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.784963    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5f043733-4943-476f-bfe4-d3dab8f2ff8b" (UID: "5f043733-4943-476f-bfe4-d3dab8f2ff8b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:32:03.786598 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.786576    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn" (OuterVolumeSpecName: "kube-api-access-wqwkn") pod "5f043733-4943-476f-bfe4-d3dab8f2ff8b" (UID: "5f043733-4943-476f-bfe4-d3dab8f2ff8b"). InnerVolumeSpecName "kube-api-access-wqwkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:32:03.885994 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.885974    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqwkn\" (UniqueName: \"kubernetes.io/projected/5f043733-4943-476f-bfe4-d3dab8f2ff8b-kube-api-access-wqwkn\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:32:03.885994 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:03.885994    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f043733-4943-476f-bfe4-d3dab8f2ff8b-extensions-socket-volume\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:32:04.082656 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.082620    2564 generic.go:356] "Generic (PLEG): container finished" podID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" containerID="00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae" exitCode=0
Jul 10 20:32:04.082802 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.082666    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"
Jul 10 20:32:04.082802 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.082708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" event={"ID":"5f043733-4943-476f-bfe4-d3dab8f2ff8b","Type":"ContainerDied","Data":"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"}
Jul 10 20:32:04.082802 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.082764    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8" event={"ID":"5f043733-4943-476f-bfe4-d3dab8f2ff8b","Type":"ContainerDied","Data":"007cc5a1f525964a000eac0d4fcf87dc843c4ef9d2cc2ca11eecee710d58e24a"}
Jul 10 20:32:04.082802 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.082792    2564 scope.go:117] "RemoveContainer" containerID="00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"
Jul 10 20:32:04.090896 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.090881    2564 scope.go:117] "RemoveContainer" containerID="00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"
Jul 10 20:32:04.091145 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:32:04.091127    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae\": container with ID starting with 00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae not found: ID does not exist" containerID="00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"
Jul 10 20:32:04.091231 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.091154    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae"} err="failed to get container status \"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae\": rpc error: code = NotFound desc = could not find container \"00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae\": container with ID starting with 00bdb89d9501731b7e3b268f72a6b8d2ce81999a6b8d63047a62d3e5b11711ae not found: ID does not exist"
Jul 10 20:32:04.104912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.104881    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:32:04.108437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.108408    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tvxf8"]
Jul 10 20:32:04.784269 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:04.784238    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" path="/var/lib/kubelet/pods/5f043733-4943-476f-bfe4-d3dab8f2ff8b/volumes"
Jul 10 20:32:09.062816 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:09.062785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-th8zc"
Jul 10 20:32:21.780119 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.780087    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:21.780580 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.780565    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" containerName="manager"
Jul 10 20:32:21.780630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.780584    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" containerName="manager"
Jul 10 20:32:21.780671 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.780661    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f043733-4943-476f-bfe4-d3dab8f2ff8b" containerName="manager"
Jul 10 20:32:21.979458 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.979423    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:21.979628 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.979480    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:21.979628 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.979569    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:21.982296 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:21.982275    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 10 20:32:22.122326 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.122296    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.122457 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.122351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qjg\" (UniqueName: \"kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.223016 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.222987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.223158 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.223030    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qjg\" (UniqueName: \"kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.223703 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.223684    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.232826 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.232803    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qjg\" (UniqueName: \"kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg\") pod \"limitador-limitador-64c94fd64f-8mqd2\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.280717 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.280685    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:32:22.289300 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.289275    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:22.323002 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.322968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:32:22.323173 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.323102    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.325332 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.325307    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:32:22.424034 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.424001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bt6b\" (UniqueName: \"kubernetes.io/projected/3d50bb32-67fb-4525-bad7-7f2f9b32549c-kube-api-access-4bt6b\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.424180 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.424045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d50bb32-67fb-4525-bad7-7f2f9b32549c-config-file\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.458051 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.458024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:22.460672 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:32:22.460642    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e38309_0bc5_40b8_9871_4c31132f409a.slice/crio-e519a90a59a3c48c2e9b2bc21e1d8252522df2bd079c6acfe9e4f648b7c62b78 WatchSource:0}: Error finding container e519a90a59a3c48c2e9b2bc21e1d8252522df2bd079c6acfe9e4f648b7c62b78: Status 404 returned error can't find the container with id e519a90a59a3c48c2e9b2bc21e1d8252522df2bd079c6acfe9e4f648b7c62b78
Jul 10 20:32:22.524900 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.524871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bt6b\" (UniqueName: \"kubernetes.io/projected/3d50bb32-67fb-4525-bad7-7f2f9b32549c-kube-api-access-4bt6b\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.525043 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.524914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d50bb32-67fb-4525-bad7-7f2f9b32549c-config-file\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.525539 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.525520    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3d50bb32-67fb-4525-bad7-7f2f9b32549c-config-file\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.533384 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.533359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bt6b\" (UniqueName: \"kubernetes.io/projected/3d50bb32-67fb-4525-bad7-7f2f9b32549c-kube-api-access-4bt6b\") pod \"limitador-limitador-54d6d988d5-ngphh\" (UID: \"3d50bb32-67fb-4525-bad7-7f2f9b32549c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.634508 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.634415    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:22.808869 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:22.808558    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:32:23.168548 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:23.168506    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh" event={"ID":"3d50bb32-67fb-4525-bad7-7f2f9b32549c","Type":"ContainerStarted","Data":"0a098edad7d8afec04e336517f22689185b10a4696d7b45109222f33a514e949"}
Jul 10 20:32:23.169612 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:23.169580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" event={"ID":"e2e38309-0bc5-40b8-9871-4c31132f409a","Type":"ContainerStarted","Data":"e519a90a59a3c48c2e9b2bc21e1d8252522df2bd079c6acfe9e4f648b7c62b78"}
Jul 10 20:32:28.196523 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.196474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh" event={"ID":"3d50bb32-67fb-4525-bad7-7f2f9b32549c","Type":"ContainerStarted","Data":"aa2a07d4b3e4a128011746b9aa32b8a54507b0dd8945cc65858f29a9bbd62903"}
Jul 10 20:32:28.197031 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.196615    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:28.198147 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.198126    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" event={"ID":"e2e38309-0bc5-40b8-9871-4c31132f409a","Type":"ContainerStarted","Data":"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"}
Jul 10 20:32:28.198247 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.198205    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:28.213563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.213517    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh" podStartSLOduration=1.392315436 podStartE2EDuration="6.213501927s" totalImagesPullingTime="4.821186491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:32:28.21140523 +0000 UTC m=+672.111710472" watchObservedRunningTime="2026-07-10 20:32:28.213501927 +0000 UTC m=+672.113807136"
Jul 10 20:32:28.226461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:28.226421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" podStartSLOduration=1.982573483 podStartE2EDuration="7.226404885s" totalImagesPullingTime="5.243831402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:32:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:32:28.225198092 +0000 UTC m=+672.125503310" watchObservedRunningTime="2026-07-10 20:32:28.226404885 +0000 UTC m=+672.126710130"
Jul 10 20:32:39.203824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.203786    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:39.204265 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.203854    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-ngphh"
Jul 10 20:32:39.263577 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.263552    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:39.263725 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.263686    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" podUID="e2e38309-0bc5-40b8-9871-4c31132f409a" containerName="limitador" containerID="cri-o://5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602" gracePeriod=30
Jul 10 20:32:39.804634 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.804606    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:39.852369 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.852312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8qjg\" (UniqueName: \"kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg\") pod \"e2e38309-0bc5-40b8-9871-4c31132f409a\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") "
Jul 10 20:32:39.852369 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.852342    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file\") pod \"e2e38309-0bc5-40b8-9871-4c31132f409a\" (UID: \"e2e38309-0bc5-40b8-9871-4c31132f409a\") "
Jul 10 20:32:39.852730 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.852711    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file" (OuterVolumeSpecName: "config-file") pod "e2e38309-0bc5-40b8-9871-4c31132f409a" (UID: "e2e38309-0bc5-40b8-9871-4c31132f409a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:32:39.854276 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.854251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg" (OuterVolumeSpecName: "kube-api-access-k8qjg") pod "e2e38309-0bc5-40b8-9871-4c31132f409a" (UID: "e2e38309-0bc5-40b8-9871-4c31132f409a"). InnerVolumeSpecName "kube-api-access-k8qjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:32:39.953721 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.953699    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qjg\" (UniqueName: \"kubernetes.io/projected/e2e38309-0bc5-40b8-9871-4c31132f409a-kube-api-access-k8qjg\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:32:39.953721 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:39.953720    2564 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2e38309-0bc5-40b8-9871-4c31132f409a-config-file\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:32:40.247022 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.246938    2564 generic.go:356] "Generic (PLEG): container finished" podID="e2e38309-0bc5-40b8-9871-4c31132f409a" containerID="5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602" exitCode=0
Jul 10 20:32:40.247022 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.246987    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" event={"ID":"e2e38309-0bc5-40b8-9871-4c31132f409a","Type":"ContainerDied","Data":"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"}
Jul 10 20:32:40.247022 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.247024    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2" event={"ID":"e2e38309-0bc5-40b8-9871-4c31132f409a","Type":"ContainerDied","Data":"e519a90a59a3c48c2e9b2bc21e1d8252522df2bd079c6acfe9e4f648b7c62b78"}
Jul 10 20:32:40.247537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.247045    2564 scope.go:117] "RemoveContainer" containerID="5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"
Jul 10 20:32:40.247537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.246986    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"
Jul 10 20:32:40.255308 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.255285    2564 scope.go:117] "RemoveContainer" containerID="5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"
Jul 10 20:32:40.255594 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:32:40.255559    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602\": container with ID starting with 5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602 not found: ID does not exist" containerID="5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"
Jul 10 20:32:40.255664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.255601    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602"} err="failed to get container status \"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602\": rpc error: code = NotFound desc = could not find container \"5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602\": container with ID starting with 5dcba188aa57b468169c5925a347b66057eb118a30cab65cc9cb6544042ad602 not found: ID does not exist"
Jul 10 20:32:40.268429 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.268406    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:40.271512 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.271470    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-8mqd2"]
Jul 10 20:32:40.783892 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:40.783858    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e38309-0bc5-40b8-9871-4c31132f409a" path="/var/lib/kubelet/pods/e2e38309-0bc5-40b8-9871-4c31132f409a/volumes"
Jul 10 20:32:43.301808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.301778    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24sds"]
Jul 10 20:32:43.302151 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.302056    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e38309-0bc5-40b8-9871-4c31132f409a" containerName="limitador"
Jul 10 20:32:43.302151 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.302067    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e38309-0bc5-40b8-9871-4c31132f409a" containerName="limitador"
Jul 10 20:32:43.302151 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.302130    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e38309-0bc5-40b8-9871-4c31132f409a" containerName="limitador"
Jul 10 20:32:43.304950 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.304933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.307685 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.307662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 10 20:32:43.307878 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.307861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zgj6l\""
Jul 10 20:32:43.307939 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.307893    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 10 20:32:43.309070 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.309047    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 10 20:32:43.314719 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.314698    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24sds"]
Jul 10 20:32:43.378848 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.378817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ba88514-3230-4b67-a8f3-24c53e7127f6-data\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.378989 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.378883    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4cj4\" (UniqueName: \"kubernetes.io/projected/5ba88514-3230-4b67-a8f3-24c53e7127f6-kube-api-access-b4cj4\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.480077 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.480043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ba88514-3230-4b67-a8f3-24c53e7127f6-data\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.480215 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.480103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4cj4\" (UniqueName: \"kubernetes.io/projected/5ba88514-3230-4b67-a8f3-24c53e7127f6-kube-api-access-b4cj4\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.480446 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.480425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5ba88514-3230-4b67-a8f3-24c53e7127f6-data\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.488115 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.488092    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4cj4\" (UniqueName: \"kubernetes.io/projected/5ba88514-3230-4b67-a8f3-24c53e7127f6-kube-api-access-b4cj4\") pod \"postgres-69fb4594cf-24sds\" (UID: \"5ba88514-3230-4b67-a8f3-24c53e7127f6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.616477 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.616452    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:43.738554 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:43.738529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-24sds"]
Jul 10 20:32:43.740405 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:32:43.740372    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba88514_3230_4b67_a8f3_24c53e7127f6.slice/crio-3f9d00887c4b94c3f5c3955ba3e9f10c8a484bd6e4ada8db9ecf899cfe3381fd WatchSource:0}: Error finding container 3f9d00887c4b94c3f5c3955ba3e9f10c8a484bd6e4ada8db9ecf899cfe3381fd: Status 404 returned error can't find the container with id 3f9d00887c4b94c3f5c3955ba3e9f10c8a484bd6e4ada8db9ecf899cfe3381fd
Jul 10 20:32:44.265754 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:44.265725    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds" event={"ID":"5ba88514-3230-4b67-a8f3-24c53e7127f6","Type":"ContainerStarted","Data":"3f9d00887c4b94c3f5c3955ba3e9f10c8a484bd6e4ada8db9ecf899cfe3381fd"}
Jul 10 20:32:49.292608 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:49.292573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds" event={"ID":"5ba88514-3230-4b67-a8f3-24c53e7127f6","Type":"ContainerStarted","Data":"05ef256d4ac962464d6a76cb5f10ad48be9e0885a60ffdee34372caa11068cb0"}
Jul 10 20:32:49.292999 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:49.292657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:32:49.308321 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:49.308278    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds" podStartSLOduration=1.537428494 podStartE2EDuration="6.308266782s" totalImagesPullingTime="4.770838288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:32:49.307422705 +0000 UTC m=+693.207727949" watchObservedRunningTime="2026-07-10 20:32:49.308266782 +0000 UTC m=+693.208572013"
Jul 10 20:32:55.325476 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:32:55.325450    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-24sds"
Jul 10 20:33:02.200565 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.200529    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:02.315675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.315643    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:02.315822 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.315753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.319876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.319843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 10 20:33:02.319876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.319858    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvflm\""
Jul 10 20:33:02.319876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.319865    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 10 20:33:02.369920 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.369890    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:33:02.400547 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.400517    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:33:02.400673 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.400624    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.418178 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.418152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvwf\" (UniqueName: \"kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.418292 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.418232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.518778 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.518717    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9bcw\" (UniqueName: \"kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.518778 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.518755    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.518936 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.518804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.518936 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.518846    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvwf\" (UniqueName: \"kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.521020 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.521000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.527775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.527755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvwf\" (UniqueName: \"kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf\") pod \"maas-controller-56d89459bc-jwfwz\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") " pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.620102 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.620072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9bcw\" (UniqueName: \"kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.620238 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.620132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.622319 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.622296    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.626104 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.626084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:02.628616 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.628597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9bcw\" (UniqueName: \"kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw\") pod \"maas-controller-6789fdfff6-bjztv\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") " pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.710941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.710906    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:02.807745 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.807716    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:02.810782 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:33:02.810756    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cfaa13_f56f_4cdf_9c40_f9f907ac2856.slice/crio-93450bfabf1c117937686bee230bdfee36f8f85cf9e2c45f937a79fec58614f9 WatchSource:0}: Error finding container 93450bfabf1c117937686bee230bdfee36f8f85cf9e2c45f937a79fec58614f9: Status 404 returned error can't find the container with id 93450bfabf1c117937686bee230bdfee36f8f85cf9e2c45f937a79fec58614f9
Jul 10 20:33:02.868953 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:02.868930    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:33:02.870761 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:33:02.870732    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95482986_4ea0_451b_bd47_c5c572b665a5.slice/crio-edb6a0a6277a6fa0732b0dec6588f78a557ec29c18b626eafc2ee15f3cc884dd WatchSource:0}: Error finding container edb6a0a6277a6fa0732b0dec6588f78a557ec29c18b626eafc2ee15f3cc884dd: Status 404 returned error can't find the container with id edb6a0a6277a6fa0732b0dec6588f78a557ec29c18b626eafc2ee15f3cc884dd
Jul 10 20:33:03.355756 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:03.355718    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d89459bc-jwfwz" event={"ID":"28cfaa13-f56f-4cdf-9c40-f9f907ac2856","Type":"ContainerStarted","Data":"93450bfabf1c117937686bee230bdfee36f8f85cf9e2c45f937a79fec58614f9"}
Jul 10 20:33:03.357068 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:03.357038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-bjztv" event={"ID":"95482986-4ea0-451b-bd47-c5c572b665a5","Type":"ContainerStarted","Data":"edb6a0a6277a6fa0732b0dec6588f78a557ec29c18b626eafc2ee15f3cc884dd"}
Jul 10 20:33:06.376156 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.376118    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d89459bc-jwfwz" event={"ID":"28cfaa13-f56f-4cdf-9c40-f9f907ac2856","Type":"ContainerStarted","Data":"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"}
Jul 10 20:33:06.376683 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.376219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:06.377769 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.377747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-bjztv" event={"ID":"95482986-4ea0-451b-bd47-c5c572b665a5","Type":"ContainerStarted","Data":"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"}
Jul 10 20:33:06.377876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.377799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:06.396193 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.396144    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56d89459bc-jwfwz" podStartSLOduration=1.701651741 podStartE2EDuration="4.396130037s" totalImagesPullingTime="2.694478296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:33:06.393746213 +0000 UTC m=+710.294051458" watchObservedRunningTime="2026-07-10 20:33:06.396130037 +0000 UTC m=+710.296435256"
Jul 10 20:33:06.418549 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:06.418483    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6789fdfff6-bjztv" podStartSLOduration=1.791093358 podStartE2EDuration="4.418469258s" totalImagesPullingTime="2.6273759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:33:06.417135885 +0000 UTC m=+710.317441115" watchObservedRunningTime="2026-07-10 20:33:06.418469258 +0000 UTC m=+710.318774500"
Jul 10 20:33:17.392587 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.392554    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:17.393008 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.392941    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:33:17.456151 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.456123    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:17.456352 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.456328    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56d89459bc-jwfwz" podUID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" containerName="manager" containerID="cri-o://4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311" gracePeriod=30
Jul 10 20:33:17.701334 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.701311    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:17.844706 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.844675    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfvwf\" (UniqueName: \"kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf\") pod \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") "
Jul 10 20:33:17.844878 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.844806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert\") pod \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\" (UID: \"28cfaa13-f56f-4cdf-9c40-f9f907ac2856\") "
Jul 10 20:33:17.847063 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.847033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "28cfaa13-f56f-4cdf-9c40-f9f907ac2856" (UID: "28cfaa13-f56f-4cdf-9c40-f9f907ac2856"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:33:17.847175 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.847032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf" (OuterVolumeSpecName: "kube-api-access-xfvwf") pod "28cfaa13-f56f-4cdf-9c40-f9f907ac2856" (UID: "28cfaa13-f56f-4cdf-9c40-f9f907ac2856"). InnerVolumeSpecName "kube-api-access-xfvwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:33:17.945674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.945621    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvwf\" (UniqueName: \"kubernetes.io/projected/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-kube-api-access-xfvwf\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:33:17.945674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:17.945643    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28cfaa13-f56f-4cdf-9c40-f9f907ac2856-webhook-cert\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:33:18.436149 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.436114    2564 generic.go:356] "Generic (PLEG): container finished" podID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" containerID="4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311" exitCode=0
Jul 10 20:33:18.436529 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.436174    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56d89459bc-jwfwz"
Jul 10 20:33:18.436529 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.436203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d89459bc-jwfwz" event={"ID":"28cfaa13-f56f-4cdf-9c40-f9f907ac2856","Type":"ContainerDied","Data":"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"}
Jul 10 20:33:18.436529 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.436288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56d89459bc-jwfwz" event={"ID":"28cfaa13-f56f-4cdf-9c40-f9f907ac2856","Type":"ContainerDied","Data":"93450bfabf1c117937686bee230bdfee36f8f85cf9e2c45f937a79fec58614f9"}
Jul 10 20:33:18.436529 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.436317    2564 scope.go:117] "RemoveContainer" containerID="4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"
Jul 10 20:33:18.444717 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.444703    2564 scope.go:117] "RemoveContainer" containerID="4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"
Jul 10 20:33:18.444966 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:33:18.444949    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311\": container with ID starting with 4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311 not found: ID does not exist" containerID="4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"
Jul 10 20:33:18.445008 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.444974    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311"} err="failed to get container status \"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311\": rpc error: code = NotFound desc = could not find container \"4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311\": container with ID starting with 4598a93206b7ef3559ffb9bec830845b48e4f9d95315274ae4bb2b19a6af8311 not found: ID does not exist"
Jul 10 20:33:18.458478 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.458455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:18.464552 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.464531    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56d89459bc-jwfwz"]
Jul 10 20:33:18.784003 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:18.783936    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" path="/var/lib/kubelet/pods/28cfaa13-f56f-4cdf-9c40-f9f907ac2856/volumes"
Jul 10 20:33:36.318279 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.318244    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"]
Jul 10 20:33:36.318894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.318752    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" containerName="manager"
Jul 10 20:33:36.318894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.318773    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" containerName="manager"
Jul 10 20:33:36.318894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.318857    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28cfaa13-f56f-4cdf-9c40-f9f907ac2856" containerName="manager"
Jul 10 20:33:36.323167 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.323147    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.327061 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.327031    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cdfgh\""
Jul 10 20:33:36.327061 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.327037    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 10 20:33:36.332228 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.332201    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"]
Jul 10 20:33:36.487506 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.487471    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvh2b\" (UniqueName: \"kubernetes.io/projected/506bf005-d114-4857-aeb9-0c7f34aad58c-kube-api-access-nvh2b\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.487667 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.487538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf005-d114-4857-aeb9-0c7f34aad58c-openshift-service-ca\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.487667 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.487596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf005-d114-4857-aeb9-0c7f34aad58c-maas-api-tls\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.588050 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.587961    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf005-d114-4857-aeb9-0c7f34aad58c-maas-api-tls\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.588050 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.588015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvh2b\" (UniqueName: \"kubernetes.io/projected/506bf005-d114-4857-aeb9-0c7f34aad58c-kube-api-access-nvh2b\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.588050 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.588043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf005-d114-4857-aeb9-0c7f34aad58c-openshift-service-ca\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.588742 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.588722    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/506bf005-d114-4857-aeb9-0c7f34aad58c-openshift-service-ca\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.590325 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.590306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/506bf005-d114-4857-aeb9-0c7f34aad58c-maas-api-tls\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.595707 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.595690    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvh2b\" (UniqueName: \"kubernetes.io/projected/506bf005-d114-4857-aeb9-0c7f34aad58c-kube-api-access-nvh2b\") pod \"maas-api-84fc6df8d5-k2j6v\" (UID: \"506bf005-d114-4857-aeb9-0c7f34aad58c\") " pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.633646 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.633617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:36.769084 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.769063    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"]
Jul 10 20:33:36.771110 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:33:36.771080    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506bf005_d114_4857_aeb9_0c7f34aad58c.slice/crio-35264545f6977f0144f8cdceee48b952e26f17c2048c86f5ef35ebab2cf74f78 WatchSource:0}: Error finding container 35264545f6977f0144f8cdceee48b952e26f17c2048c86f5ef35ebab2cf74f78: Status 404 returned error can't find the container with id 35264545f6977f0144f8cdceee48b952e26f17c2048c86f5ef35ebab2cf74f78
Jul 10 20:33:36.968698 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:36.968612    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:33:37.523353 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:37.523313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v" event={"ID":"506bf005-d114-4857-aeb9-0c7f34aad58c","Type":"ContainerStarted","Data":"35264545f6977f0144f8cdceee48b952e26f17c2048c86f5ef35ebab2cf74f78"}
Jul 10 20:33:39.533667 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:39.533627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v" event={"ID":"506bf005-d114-4857-aeb9-0c7f34aad58c","Type":"ContainerStarted","Data":"5fca930325e0098f18b26de3eda707705eefb01556fa776e79a6794db58b2f44"}
Jul 10 20:33:39.534117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:39.533735    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:39.550579 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:39.550535    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v" podStartSLOduration=1.78163951 podStartE2EDuration="3.550522008s" totalImagesPullingTime="1.768882498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:33:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:33:39.54932559 +0000 UTC m=+743.449630820" watchObservedRunningTime="2026-07-10 20:33:39.550522008 +0000 UTC m=+743.450827227"
Jul 10 20:33:45.543709 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:45.543674    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84fc6df8d5-k2j6v"
Jul 10 20:33:56.317641 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:33:56.317609    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:34:03.813041 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.813011    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"]
Jul 10 20:34:03.816761 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.816740    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.821057 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.821037    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tmx44\""
Jul 10 20:34:03.821057 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.821045    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 10 20:34:03.821228 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.821084    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 10 20:34:03.821228 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.821045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 10 20:34:03.827932 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.827903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"]
Jul 10 20:34:03.905178 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.905178 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.905349 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905198    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.905349 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.905349 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:03.905349 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:03.905304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fck9h\" (UniqueName: \"kubernetes.io/projected/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-kube-api-access-fck9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.006644 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.006797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.006797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fck9h\" (UniqueName: \"kubernetes.io/projected/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-kube-api-access-fck9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.006797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.006797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.007011 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.006890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.007069 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.007018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.007186 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.007154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.007307 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.007283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.008948 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.008931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.009108 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.009093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.014668 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.014649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fck9h\" (UniqueName: \"kubernetes.io/projected/4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9-kube-api-access-fck9h\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dln2j\" (UID: \"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.128168 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.128098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:04.256450 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.256280    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"]
Jul 10 20:34:04.259521 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:34:04.259471    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6ed507_5e7d_49a4_a3e9_fc3c533da0e9.slice/crio-73223eb1988bc1c9e8264bf5b3bd55075b136eaeb8172e4d6f90615f937758ae WatchSource:0}: Error finding container 73223eb1988bc1c9e8264bf5b3bd55075b136eaeb8172e4d6f90615f937758ae: Status 404 returned error can't find the container with id 73223eb1988bc1c9e8264bf5b3bd55075b136eaeb8172e4d6f90615f937758ae
Jul 10 20:34:04.531758 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.531678    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:34:04.657045 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:04.656966    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j" event={"ID":"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9","Type":"ContainerStarted","Data":"73223eb1988bc1c9e8264bf5b3bd55075b136eaeb8172e4d6f90615f937758ae"}
Jul 10 20:34:05.663915 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:05.663885    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j" event={"ID":"4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9","Type":"ContainerStarted","Data":"4c3a7c08647a64898340b9d3afc9824cee9f341866761e7dcb5dd899e15b9882"}
Jul 10 20:34:05.683506 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:05.683448    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j" podStartSLOduration=1.6753447540000002 podStartE2EDuration="2.683432192s" totalImagesPullingTime="1.008087438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:34:05.681717187 +0000 UTC m=+769.582022447" watchObservedRunningTime="2026-07-10 20:34:05.683432192 +0000 UTC m=+769.583737412"
Jul 10 20:34:10.020437 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:10.020401    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:34:14.128381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:14.128347    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:14.128848 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:14.128418    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:14.141068 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:14.141046    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:14.711254 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:14.711229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dln2j"
Jul 10 20:34:15.721537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:15.721507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:34:18.212675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.212644    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"]
Jul 10 20:34:18.217367 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.217343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.220112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.220093    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 10 20:34:18.227892 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.227869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"]
Jul 10 20:34:18.317912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.317881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.317912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.317917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4470944b-fc16-47a4-b856-778e2de25dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.318144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.317944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.318144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.317998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.318144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.318021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.318144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.318070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rfn2\" (UniqueName: \"kubernetes.io/projected/4470944b-fc16-47a4-b856-778e2de25dbd-kube-api-access-5rfn2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4470944b-fc16-47a4-b856-778e2de25dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418904 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418904 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418996 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418902    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.418996 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.418942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rfn2\" (UniqueName: \"kubernetes.io/projected/4470944b-fc16-47a4-b856-778e2de25dbd-kube-api-access-5rfn2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.419264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.419233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.419409 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.419271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.419409 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.419319    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.420976 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.420959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4470944b-fc16-47a4-b856-778e2de25dbd-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.421256 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.421237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4470944b-fc16-47a4-b856-778e2de25dbd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.427747 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.427726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rfn2\" (UniqueName: \"kubernetes.io/projected/4470944b-fc16-47a4-b856-778e2de25dbd-kube-api-access-5rfn2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ddshx\" (UID: \"4470944b-fc16-47a4-b856-778e2de25dbd\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.528125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.528056    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:18.656681 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.656650    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"]
Jul 10 20:34:18.660236 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:34:18.660208    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4470944b_fc16_47a4_b856_778e2de25dbd.slice/crio-d5011898ef1497263c7a33de4cb17a2e12e2557652706496eb14f67de7a67fb2 WatchSource:0}: Error finding container d5011898ef1497263c7a33de4cb17a2e12e2557652706496eb14f67de7a67fb2: Status 404 returned error can't find the container with id d5011898ef1497263c7a33de4cb17a2e12e2557652706496eb14f67de7a67fb2
Jul 10 20:34:18.661610 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.661591    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:34:18.723714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:18.723678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx" event={"ID":"4470944b-fc16-47a4-b856-778e2de25dbd","Type":"ContainerStarted","Data":"d5011898ef1497263c7a33de4cb17a2e12e2557652706496eb14f67de7a67fb2"}
Jul 10 20:34:19.729370 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:19.729337    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx" event={"ID":"4470944b-fc16-47a4-b856-778e2de25dbd","Type":"ContainerStarted","Data":"3124d3e99aacb0d557019a96c5a55105428ed7e7e5f61c6f51ada4e1d43c44fe"}
Jul 10 20:34:19.747541 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:19.747471    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx" podStartSLOduration=1.579045286 podStartE2EDuration="1.747456328s" totalImagesPullingTime="168.411042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:34:19.746638229 +0000 UTC m=+783.646943470" watchObservedRunningTime="2026-07-10 20:34:19.747456328 +0000 UTC m=+783.647761538"
Jul 10 20:34:24.363313 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:24.363282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:34:28.529160 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:28.529128    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:28.529541 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:28.529189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:28.541407 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:28.541385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:28.784351 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:28.784280    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ddshx"
Jul 10 20:34:32.429450 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:34:32.429422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:14.930806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:14.930738    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:16.658523 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:16.658478    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:36:16.659355 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:16.659338    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:36:24.728568 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:24.728445    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:36.131241 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:36.131206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:44.125478 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:44.125441    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:55.329637 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:55.329598    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:36:57.678643 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:57.678605    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:36:57.679112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:57.678905    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6789fdfff6-bjztv" podUID="95482986-4ea0-451b-bd47-c5c572b665a5" containerName="manager" containerID="cri-o://f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe" gracePeriod=30
Jul 10 20:36:57.927649 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:57.927624    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:36:58.046058 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.045975    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9bcw\" (UniqueName: \"kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw\") pod \"95482986-4ea0-451b-bd47-c5c572b665a5\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") "
Jul 10 20:36:58.046210 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.046082    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert\") pod \"95482986-4ea0-451b-bd47-c5c572b665a5\" (UID: \"95482986-4ea0-451b-bd47-c5c572b665a5\") "
Jul 10 20:36:58.047980 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.047948    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw" (OuterVolumeSpecName: "kube-api-access-b9bcw") pod "95482986-4ea0-451b-bd47-c5c572b665a5" (UID: "95482986-4ea0-451b-bd47-c5c572b665a5"). InnerVolumeSpecName "kube-api-access-b9bcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:36:58.047980 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.047959    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95482986-4ea0-451b-bd47-c5c572b665a5" (UID: "95482986-4ea0-451b-bd47-c5c572b665a5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:36:58.147198 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.147158    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9bcw\" (UniqueName: \"kubernetes.io/projected/95482986-4ea0-451b-bd47-c5c572b665a5-kube-api-access-b9bcw\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:36:58.147198 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.147191    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95482986-4ea0-451b-bd47-c5c572b665a5-webhook-cert\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:36:58.439134 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.439098    2564 generic.go:356] "Generic (PLEG): container finished" podID="95482986-4ea0-451b-bd47-c5c572b665a5" containerID="f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe" exitCode=0
Jul 10 20:36:58.439300 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.439144    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-bjztv"
Jul 10 20:36:58.439300 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.439181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-bjztv" event={"ID":"95482986-4ea0-451b-bd47-c5c572b665a5","Type":"ContainerDied","Data":"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"}
Jul 10 20:36:58.439300 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.439238    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-bjztv" event={"ID":"95482986-4ea0-451b-bd47-c5c572b665a5","Type":"ContainerDied","Data":"edb6a0a6277a6fa0732b0dec6588f78a557ec29c18b626eafc2ee15f3cc884dd"}
Jul 10 20:36:58.439300 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.439268    2564 scope.go:117] "RemoveContainer" containerID="f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"
Jul 10 20:36:58.448178 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.448162    2564 scope.go:117] "RemoveContainer" containerID="f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"
Jul 10 20:36:58.448414 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:36:58.448396    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe\": container with ID starting with f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe not found: ID does not exist" containerID="f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"
Jul 10 20:36:58.448477 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.448422    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe"} err="failed to get container status \"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe\": rpc error: code = NotFound desc = could not find container \"f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe\": container with ID starting with f582678d62553acc820f8b541bd98baf33824071126c094e6c0e6014939c1ffe not found: ID does not exist"
Jul 10 20:36:58.459716 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.459694    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:36:58.463264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.463245    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-bjztv"]
Jul 10 20:36:58.784659 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:58.784585    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95482986-4ea0-451b-bd47-c5c572b665a5" path="/var/lib/kubelet/pods/95482986-4ea0-451b-bd47-c5c572b665a5/volumes"
Jul 10 20:36:59.256498 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.256466    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6789fdfff6-kxdbj"]
Jul 10 20:36:59.256825 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.256813    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95482986-4ea0-451b-bd47-c5c572b665a5" containerName="manager"
Jul 10 20:36:59.256871 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.256827    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="95482986-4ea0-451b-bd47-c5c572b665a5" containerName="manager"
Jul 10 20:36:59.256905 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.256887    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="95482986-4ea0-451b-bd47-c5c572b665a5" containerName="manager"
Jul 10 20:36:59.260870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.260854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.264791 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.264769    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nvflm\""
Jul 10 20:36:59.264906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.264797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 10 20:36:59.264906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.264863    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 10 20:36:59.268367 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.268121    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-kxdbj"]
Jul 10 20:36:59.358683 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.358649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540c12c3-dec8-468b-8cad-16b1da9bedea-webhook-cert\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.358854 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.358713    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rgf\" (UniqueName: \"kubernetes.io/projected/540c12c3-dec8-468b-8cad-16b1da9bedea-kube-api-access-84rgf\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.459681 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.459649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540c12c3-dec8-468b-8cad-16b1da9bedea-webhook-cert\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.459844 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.459712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rgf\" (UniqueName: \"kubernetes.io/projected/540c12c3-dec8-468b-8cad-16b1da9bedea-kube-api-access-84rgf\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.465840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.463260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/540c12c3-dec8-468b-8cad-16b1da9bedea-webhook-cert\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.468757 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.468733    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rgf\" (UniqueName: \"kubernetes.io/projected/540c12c3-dec8-468b-8cad-16b1da9bedea-kube-api-access-84rgf\") pod \"maas-controller-6789fdfff6-kxdbj\" (UID: \"540c12c3-dec8-468b-8cad-16b1da9bedea\") " pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.572252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.572225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:36:59.701013 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:36:59.700652    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6789fdfff6-kxdbj"]
Jul 10 20:36:59.703509 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:36:59.703455    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540c12c3_dec8_468b_8cad_16b1da9bedea.slice/crio-bdeaf2eda8cbae1c47b857247d7447ac6b1bd9c76a22ded388645dd2d7ba217d WatchSource:0}: Error finding container bdeaf2eda8cbae1c47b857247d7447ac6b1bd9c76a22ded388645dd2d7ba217d: Status 404 returned error can't find the container with id bdeaf2eda8cbae1c47b857247d7447ac6b1bd9c76a22ded388645dd2d7ba217d
Jul 10 20:37:00.449785 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:00.449749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-kxdbj" event={"ID":"540c12c3-dec8-468b-8cad-16b1da9bedea","Type":"ContainerStarted","Data":"ab5b2a5b7b663999384f958f3e371e1a0f22f174763250d22c4b9b764ecd2d53"}
Jul 10 20:37:00.450143 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:00.449801    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6789fdfff6-kxdbj" event={"ID":"540c12c3-dec8-468b-8cad-16b1da9bedea","Type":"ContainerStarted","Data":"bdeaf2eda8cbae1c47b857247d7447ac6b1bd9c76a22ded388645dd2d7ba217d"}
Jul 10 20:37:00.450143 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:00.449857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:37:00.466619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:00.466574    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6789fdfff6-kxdbj" podStartSLOduration=0.940856728 podStartE2EDuration="1.466559991s" totalImagesPullingTime="525.703263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:37:00.465185079 +0000 UTC m=+944.365490299" watchObservedRunningTime="2026-07-10 20:37:00.466559991 +0000 UTC m=+944.366865215"
Jul 10 20:37:11.460756 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:11.460721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6789fdfff6-kxdbj"
Jul 10 20:37:23.619732 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:23.619694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:37:35.331104 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:37:35.331069    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:38:10.423724 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:38:10.423689    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:38:19.726870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:38:19.726836    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:38:28.124161 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:38:28.124126    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:38:39.920147 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:38:39.920116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:38:51.228436 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:38:51.228404    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:39:01.523863 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:39:01.523829    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:39:12.429281 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:39:12.429199    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:39:22.718797 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:39:22.718768    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:39:44.137764 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:39:44.137735    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:39:53.930312 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:39:53.930267    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:40:16.433929 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:40:16.433897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:40:30.733924 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:40:30.733882    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:40:42.124228 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:40:42.124152    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:40:52.332434 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:40:52.332398    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:41:08.424398 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:41:08.424365    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:41:16.687641 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:41:16.687612    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:41:16.688048 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:41:16.687682    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:41:18.233458 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:41:18.233425    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:05.722387 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:05.722349    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:17.422324 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:17.422285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:26.619334 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:26.619302    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:37.322519 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:37.322469    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:49.826037 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:49.825934    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:43:59.116757 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:43:59.116724    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:44:09.323904 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:44:09.323868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:44:19.317334 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:44:19.317300    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:44:31.321637 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:44:31.321604    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:44:39.329906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:44:39.329869    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:44:50.020505 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:44:50.020454    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:45:00.138569 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.138530    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:45:00.142087 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.142062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"
Jul 10 20:45:00.144753 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.144733    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tdxg6\""
Jul 10 20:45:00.151949 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.151929    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"
Jul 10 20:45:00.160878 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.160839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:45:00.279977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.279956    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:45:00.282068 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:45:00.282043    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852d1264_b757_44f1_ab96_bbeedefd4c2a.slice/crio-eb4612b590e82aa74dee5284775eeeaf6cfdc93b2eab6a18dbaf330e0d3928a0 WatchSource:0}: Error finding container eb4612b590e82aa74dee5284775eeeaf6cfdc93b2eab6a18dbaf330e0d3928a0: Status 404 returned error can't find the container with id eb4612b590e82aa74dee5284775eeeaf6cfdc93b2eab6a18dbaf330e0d3928a0
Jul 10 20:45:00.283939 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.283923    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:45:00.567057 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:00.567026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" event={"ID":"852d1264-b757-44f1-ab96-bbeedefd4c2a","Type":"ContainerStarted","Data":"eb4612b590e82aa74dee5284775eeeaf6cfdc93b2eab6a18dbaf330e0d3928a0"}
Jul 10 20:45:11.426481 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:11.426407    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:45:20.656436 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:20.656360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" event={"ID":"852d1264-b757-44f1-ab96-bbeedefd4c2a","Type":"ContainerStarted","Data":"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"}
Jul 10 20:45:20.672105 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:20.672057    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" podStartSLOduration=0.595669626 podStartE2EDuration="20.672044429s" totalImagesPullingTime="20.076374803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:45:20.671050943 +0000 UTC m=+1444.571356186" watchObservedRunningTime="2026-07-10 20:45:20.672044429 +0000 UTC m=+1444.572349642"
Jul 10 20:45:21.327184 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:21.327141    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:45:31.521942 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:31.521905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:45:41.326043 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:41.326010    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:45:50.425053 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:45:50.425024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:46:00.009809 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:00.009768    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:46:00.010280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:00.010023    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" podUID="852d1264-b757-44f1-ab96-bbeedefd4c2a" containerName="cleanup" containerID="cri-o://b571572676c492035b53730d77a4f712204c638822230e36e926465715574605" gracePeriod=30
Jul 10 20:46:03.430912 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:03.430874    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:46:16.715854 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:16.715827    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:46:16.717955 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:16.717936    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:46:18.126945 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:18.126903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:46:27.623599 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:27.623563    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:46:30.188664 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.188635    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"
Jul 10 20:46:30.963553 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.963516    2564 generic.go:356] "Generic (PLEG): container finished" podID="852d1264-b757-44f1-ab96-bbeedefd4c2a" containerID="b571572676c492035b53730d77a4f712204c638822230e36e926465715574605" exitCode=137
Jul 10 20:46:30.963712 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.963567    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"
Jul 10 20:46:30.963712 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.963595    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" event={"ID":"852d1264-b757-44f1-ab96-bbeedefd4c2a","Type":"ContainerDied","Data":"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"}
Jul 10 20:46:30.963712 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.963665    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n" event={"ID":"852d1264-b757-44f1-ab96-bbeedefd4c2a","Type":"ContainerDied","Data":"eb4612b590e82aa74dee5284775eeeaf6cfdc93b2eab6a18dbaf330e0d3928a0"}
Jul 10 20:46:30.963712 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.963686    2564 scope.go:117] "RemoveContainer" containerID="b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"
Jul 10 20:46:30.971765 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.971750    2564 scope.go:117] "RemoveContainer" containerID="b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"
Jul 10 20:46:30.971983 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:46:30.971965    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605\": container with ID starting with b571572676c492035b53730d77a4f712204c638822230e36e926465715574605 not found: ID does not exist" containerID="b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"
Jul 10 20:46:30.972032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.971991    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605"} err="failed to get container status \"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605\": rpc error: code = NotFound desc = could not find container \"b571572676c492035b53730d77a4f712204c638822230e36e926465715574605\": container with ID starting with b571572676c492035b53730d77a4f712204c638822230e36e926465715574605 not found: ID does not exist"
Jul 10 20:46:30.980544 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.980520    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:46:30.982799 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:30.982780    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728605-nwh7n"]
Jul 10 20:46:32.784723 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:32.784690    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="852d1264-b757-44f1-ab96-bbeedefd4c2a" path="/var/lib/kubelet/pods/852d1264-b757-44f1-ab96-bbeedefd4c2a/volumes"
Jul 10 20:46:41.237041 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:41.236962    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:46:52.522242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:46:52.522206    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:47:04.430887 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:47:04.430852    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:47:20.638808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:47:20.638774    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:47:33.123429 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:47:33.123389    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:47:43.322637 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:47:43.322608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:47:50.022285 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:47:50.022244    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:48:05.228650 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:48:05.228612    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:48:17.227072 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:48:17.227001    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:48:28.428096 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:48:28.428062    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:48:37.925342 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:48:37.925308    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:48:48.027505 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:48:48.027455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:49:00.424637 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:00.424606    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:49:06.535033 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.534995    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:06.535401 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.535363    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="852d1264-b757-44f1-ab96-bbeedefd4c2a" containerName="cleanup"
Jul 10 20:49:06.535401 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.535374    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="852d1264-b757-44f1-ab96-bbeedefd4c2a" containerName="cleanup"
Jul 10 20:49:06.535471 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.535442    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="852d1264-b757-44f1-ab96-bbeedefd4c2a" containerName="cleanup"
Jul 10 20:49:06.538450 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.538433    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.541197 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.541177    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-34b69dd3-serving-cert\""
Jul 10 20:49:06.548875 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.548848    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:06.591213 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.591188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.591301 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.591221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.591301 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.591256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqlt2\" (UniqueName: \"kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.692313 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.692284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.692435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.692329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.692435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.692380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqlt2\" (UniqueName: \"kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.692586 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:06.692507    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-serving-cert: secret "maas-api-e2e-ait-34b69dd3-serving-cert" not found
Jul 10 20:49:06.692586 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:06.692573    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls podName:558e3c5a-5740-43c2-8988-e662582d4577 nodeName:}" failed. No retries permitted until 2026-07-10 20:49:07.192557789 +0000 UTC m=+1671.092862985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls") pod "maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" (UID: "558e3c5a-5740-43c2-8988-e662582d4577") : secret "maas-api-e2e-ait-34b69dd3-serving-cert" not found
Jul 10 20:49:06.692925 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.692906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:06.703130 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:06.703102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqlt2\" (UniqueName: \"kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:07.196933 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:07.196899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:07.199609 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:07.199587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") pod \"maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:07.449405 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:07.449336    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:07.791576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:07.791428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:07.794459 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:49:07.794430    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558e3c5a_5740_43c2_8988_e662582d4577.slice/crio-44cce9369b581ffeb1ead0176f9d4d2e9b4df596d1e9b456b6bc01da13d57fa0 WatchSource:0}: Error finding container 44cce9369b581ffeb1ead0176f9d4d2e9b4df596d1e9b456b6bc01da13d57fa0: Status 404 returned error can't find the container with id 44cce9369b581ffeb1ead0176f9d4d2e9b4df596d1e9b456b6bc01da13d57fa0
Jul 10 20:49:08.664703 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:08.664668    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" event={"ID":"558e3c5a-5740-43c2-8988-e662582d4577","Type":"ContainerStarted","Data":"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"}
Jul 10 20:49:08.664922 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:08.664717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" event={"ID":"558e3c5a-5740-43c2-8988-e662582d4577","Type":"ContainerStarted","Data":"44cce9369b581ffeb1ead0176f9d4d2e9b4df596d1e9b456b6bc01da13d57fa0"}
Jul 10 20:49:08.664922 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:08.664773    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:08.684963 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:08.684921    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" podStartSLOduration=2.263826677 podStartE2EDuration="2.684908472s" totalImagesPullingTime="421.081795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:08.684190699 +0000 UTC m=+1672.584495918" watchObservedRunningTime="2026-07-10 20:49:08.684908472 +0000 UTC m=+1672.585213691"
Jul 10 20:49:12.061656 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.061617    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:12.062103 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.061838    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" podUID="558e3c5a-5740-43c2-8988-e662582d4577" containerName="maas-api" containerID="cri-o://03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63" gracePeriod=30
Jul 10 20:49:12.302702 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.302673    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:12.334178 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.334153    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqlt2\" (UniqueName: \"kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2\") pod \"558e3c5a-5740-43c2-8988-e662582d4577\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") "
Jul 10 20:49:12.334299 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.334250    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca\") pod \"558e3c5a-5740-43c2-8988-e662582d4577\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") "
Jul 10 20:49:12.334299 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.334281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") pod \"558e3c5a-5740-43c2-8988-e662582d4577\" (UID: \"558e3c5a-5740-43c2-8988-e662582d4577\") "
Jul 10 20:49:12.334587 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.334565    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "558e3c5a-5740-43c2-8988-e662582d4577" (UID: "558e3c5a-5740-43c2-8988-e662582d4577"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:49:12.336292 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.336271    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2" (OuterVolumeSpecName: "kube-api-access-nqlt2") pod "558e3c5a-5740-43c2-8988-e662582d4577" (UID: "558e3c5a-5740-43c2-8988-e662582d4577"). InnerVolumeSpecName "kube-api-access-nqlt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:49:12.336381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.336353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "558e3c5a-5740-43c2-8988-e662582d4577" (UID: "558e3c5a-5740-43c2-8988-e662582d4577"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:49:12.435407 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.435389    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqlt2\" (UniqueName: \"kubernetes.io/projected/558e3c5a-5740-43c2-8988-e662582d4577-kube-api-access-nqlt2\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:12.435407 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.435408    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/558e3c5a-5740-43c2-8988-e662582d4577-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:12.435538 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.435417    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/558e3c5a-5740-43c2-8988-e662582d4577-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:12.684839 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.684765    2564 generic.go:356] "Generic (PLEG): container finished" podID="558e3c5a-5740-43c2-8988-e662582d4577" containerID="03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63" exitCode=0
Jul 10 20:49:12.684839 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.684810    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"
Jul 10 20:49:12.684988 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.684857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" event={"ID":"558e3c5a-5740-43c2-8988-e662582d4577","Type":"ContainerDied","Data":"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"}
Jul 10 20:49:12.684988 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.684903    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q" event={"ID":"558e3c5a-5740-43c2-8988-e662582d4577","Type":"ContainerDied","Data":"44cce9369b581ffeb1ead0176f9d4d2e9b4df596d1e9b456b6bc01da13d57fa0"}
Jul 10 20:49:12.684988 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.684924    2564 scope.go:117] "RemoveContainer" containerID="03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"
Jul 10 20:49:12.694181 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.694154    2564 scope.go:117] "RemoveContainer" containerID="03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"
Jul 10 20:49:12.694459 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:12.694441    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63\": container with ID starting with 03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63 not found: ID does not exist" containerID="03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"
Jul 10 20:49:12.694531 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.694467    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63"} err="failed to get container status \"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63\": rpc error: code = NotFound desc = could not find container \"03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63\": container with ID starting with 03373a909188be0ad351e08876a145f704fc9a34992f29023b55390f593ffa63 not found: ID does not exist"
Jul 10 20:49:12.720955 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.720928    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:12.728477 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.728452    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-34b69dd3-648c86c989-c6j6q"]
Jul 10 20:49:12.784983 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:12.784956    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="558e3c5a-5740-43c2-8988-e662582d4577" path="/var/lib/kubelet/pods/558e3c5a-5740-43c2-8988-e662582d4577/volumes"
Jul 10 20:49:19.894703 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.894673    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:19.895059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.895006    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="558e3c5a-5740-43c2-8988-e662582d4577" containerName="maas-api"
Jul 10 20:49:19.895059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.895018    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="558e3c5a-5740-43c2-8988-e662582d4577" containerName="maas-api"
Jul 10 20:49:19.895127 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.895083    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="558e3c5a-5740-43c2-8988-e662582d4577" containerName="maas-api"
Jul 10 20:49:19.898096 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.898076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:19.902040 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.902013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-412320a5-serving-cert\""
Jul 10 20:49:19.918241 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.918215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:19.986431 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.986404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:19.986574 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.986440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:19.986574 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:19.986472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtmh\" (UniqueName: \"kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.087442 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.087418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtmh\" (UniqueName: \"kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.087565 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.087463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.087565 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.087505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.087638 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:20.087583    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-serving-cert: secret "maas-api-e2e-ait-412320a5-serving-cert" not found
Jul 10 20:49:20.087669 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:20.087659    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls podName:7a29261b-4acb-4fd9-8dd3-4aad96f98c27 nodeName:}" failed. No retries permitted until 2026-07-10 20:49:20.587638919 +0000 UTC m=+1684.487944134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls") pod "maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" (UID: "7a29261b-4acb-4fd9-8dd3-4aad96f98c27") : secret "maas-api-e2e-ait-412320a5-serving-cert" not found
Jul 10 20:49:20.088100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.088083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.096644 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.096623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtmh\" (UniqueName: \"kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.591457 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.591425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.593673 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.593654    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") pod \"maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.809198 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.809167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:20.945185 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:20.945163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:20.947446 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:49:20.947425    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a29261b_4acb_4fd9_8dd3_4aad96f98c27.slice/crio-e6033c1dcbde75f3a379c365397432819a0773e12f9a6f4d904238cd1a99f8b9 WatchSource:0}: Error finding container e6033c1dcbde75f3a379c365397432819a0773e12f9a6f4d904238cd1a99f8b9: Status 404 returned error can't find the container with id e6033c1dcbde75f3a379c365397432819a0773e12f9a6f4d904238cd1a99f8b9
Jul 10 20:49:21.732393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:21.732307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" event={"ID":"7a29261b-4acb-4fd9-8dd3-4aad96f98c27","Type":"ContainerStarted","Data":"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"}
Jul 10 20:49:21.732393 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:21.732366    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" event={"ID":"7a29261b-4acb-4fd9-8dd3-4aad96f98c27","Type":"ContainerStarted","Data":"e6033c1dcbde75f3a379c365397432819a0773e12f9a6f4d904238cd1a99f8b9"}
Jul 10 20:49:21.732590 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:21.732408    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:21.762921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:21.762882    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" podStartSLOduration=2.391895484 podStartE2EDuration="2.762868909s" totalImagesPullingTime="370.973425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:21.761786643 +0000 UTC m=+1685.662091862" watchObservedRunningTime="2026-07-10 20:49:21.762868909 +0000 UTC m=+1685.663174128"
Jul 10 20:49:23.342083 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:23.342048    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:49:25.316559 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.316524    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:25.317006 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.316793    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" podUID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" containerName="maas-api" containerID="cri-o://7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430" gracePeriod=30
Jul 10 20:49:25.563558 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.563528    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:25.631916 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.631852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") pod \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") "
Jul 10 20:49:25.631916 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.631909    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvtmh\" (UniqueName: \"kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh\") pod \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") "
Jul 10 20:49:25.632133 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.631973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca\") pod \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\" (UID: \"7a29261b-4acb-4fd9-8dd3-4aad96f98c27\") "
Jul 10 20:49:25.632325 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.632301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a29261b-4acb-4fd9-8dd3-4aad96f98c27" (UID: "7a29261b-4acb-4fd9-8dd3-4aad96f98c27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:49:25.633814 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.633785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh" (OuterVolumeSpecName: "kube-api-access-vvtmh") pod "7a29261b-4acb-4fd9-8dd3-4aad96f98c27" (UID: "7a29261b-4acb-4fd9-8dd3-4aad96f98c27"). InnerVolumeSpecName "kube-api-access-vvtmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:49:25.633814 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.633806    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a29261b-4acb-4fd9-8dd3-4aad96f98c27" (UID: "7a29261b-4acb-4fd9-8dd3-4aad96f98c27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:49:25.732960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.732935    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:25.732960 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.732958    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:25.733090 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.732968    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtmh\" (UniqueName: \"kubernetes.io/projected/7a29261b-4acb-4fd9-8dd3-4aad96f98c27-kube-api-access-vvtmh\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:25.750989 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.750964    2564 generic.go:356] "Generic (PLEG): container finished" podID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" containerID="7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430" exitCode=0
Jul 10 20:49:25.751077 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.751014    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"
Jul 10 20:49:25.751077 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.751046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" event={"ID":"7a29261b-4acb-4fd9-8dd3-4aad96f98c27","Type":"ContainerDied","Data":"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"}
Jul 10 20:49:25.751146 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.751086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t" event={"ID":"7a29261b-4acb-4fd9-8dd3-4aad96f98c27","Type":"ContainerDied","Data":"e6033c1dcbde75f3a379c365397432819a0773e12f9a6f4d904238cd1a99f8b9"}
Jul 10 20:49:25.751146 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.751106    2564 scope.go:117] "RemoveContainer" containerID="7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"
Jul 10 20:49:25.759750 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.759729    2564 scope.go:117] "RemoveContainer" containerID="7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"
Jul 10 20:49:25.759991 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:25.759961    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430\": container with ID starting with 7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430 not found: ID does not exist" containerID="7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"
Jul 10 20:49:25.760052 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.759994    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430"} err="failed to get container status \"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430\": rpc error: code = NotFound desc = could not find container \"7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430\": container with ID starting with 7ad0531d00a042d4f3f7d618f253306f2bfec7973d3f89ce0db84ebc5b899430 not found: ID does not exist"
Jul 10 20:49:25.774124 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.774104    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:25.781753 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:25.781733    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-412320a5-5f8d46cb8b-dqr5t"]
Jul 10 20:49:26.785155 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:26.785125    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" path="/var/lib/kubelet/pods/7a29261b-4acb-4fd9-8dd3-4aad96f98c27/volumes"
Jul 10 20:49:33.329252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:33.329219    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:49:38.550835 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.550763    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:38.551153 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.551110    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" containerName="maas-api"
Jul 10 20:49:38.551153 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.551121    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" containerName="maas-api"
Jul 10 20:49:38.551220 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.551197    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a29261b-4acb-4fd9-8dd3-4aad96f98c27" containerName="maas-api"
Jul 10 20:49:38.555633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.555614    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.558461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.558427    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a405bf62-serving-cert\""
Jul 10 20:49:38.565320 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.565297    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:38.633744 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.633720    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qqv\" (UniqueName: \"kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.633881 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.633755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.633881 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.633828    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.734739 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.734710    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.734886 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.734751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qqv\" (UniqueName: \"kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.734886 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.734777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.735039 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:38.734994    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-serving-cert: secret "maas-api-e2e-derive-a405bf62-serving-cert" not found
Jul 10 20:49:38.735077 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:38.735072    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls podName:0e6cff9e-4b3b-4be7-bd6b-e561543e1b20 nodeName:}" failed. No retries permitted until 2026-07-10 20:49:39.235053844 +0000 UTC m=+1703.135359044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls") pod "maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" (UID: "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20") : secret "maas-api-e2e-derive-a405bf62-serving-cert" not found
Jul 10 20:49:38.735370 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.735352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:38.748925 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:38.748900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qqv\" (UniqueName: \"kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:39.239035 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:39.238998    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:39.241411 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:39.241386    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") pod \"maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:39.467545 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:39.467506    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:39.600820 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:39.600797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:39.603091 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:49:39.603060    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6cff9e_4b3b_4be7_bd6b_e561543e1b20.slice/crio-381a8e186eac42cf793b35561d9fa2eb52089f39f01004ca588e7f50d4ea8e4e WatchSource:0}: Error finding container 381a8e186eac42cf793b35561d9fa2eb52089f39f01004ca588e7f50d4ea8e4e: Status 404 returned error can't find the container with id 381a8e186eac42cf793b35561d9fa2eb52089f39f01004ca588e7f50d4ea8e4e
Jul 10 20:49:39.820283 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:39.820251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" event={"ID":"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20","Type":"ContainerStarted","Data":"381a8e186eac42cf793b35561d9fa2eb52089f39f01004ca588e7f50d4ea8e4e"}
Jul 10 20:49:40.825332 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:40.825299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" event={"ID":"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20","Type":"ContainerStarted","Data":"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"}
Jul 10 20:49:40.825717 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:40.825365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:40.844780 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:40.844696    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" podStartSLOduration=2.429325724 podStartE2EDuration="2.844680504s" totalImagesPullingTime="415.35478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:49:40.844102762 +0000 UTC m=+1704.744407998" watchObservedRunningTime="2026-07-10 20:49:40.844680504 +0000 UTC m=+1704.744985723"
Jul 10 20:49:41.173821 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:41.173740    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:49:43.316617 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.316580    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:43.317040 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.316806    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" podUID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" containerName="maas-api" containerID="cri-o://fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5" gracePeriod=30
Jul 10 20:49:43.567552 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.567468    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:43.679370 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.679340    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7qqv\" (UniqueName: \"kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv\") pod \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") "
Jul 10 20:49:43.679576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.679388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca\") pod \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") "
Jul 10 20:49:43.679576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.679437    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") pod \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\" (UID: \"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20\") "
Jul 10 20:49:43.679787 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.679762    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" (UID: "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:49:43.681447 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.681426    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv" (OuterVolumeSpecName: "kube-api-access-l7qqv") pod "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" (UID: "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20"). InnerVolumeSpecName "kube-api-access-l7qqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:49:43.681533 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.681465    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" (UID: "0e6cff9e-4b3b-4be7-bd6b-e561543e1b20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:49:43.780379 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.780336    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:43.780379 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.780374    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qqv\" (UniqueName: \"kubernetes.io/projected/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-kube-api-access-l7qqv\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:43.780379 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.780387    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:49:43.839663 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.839574    2564 generic.go:356] "Generic (PLEG): container finished" podID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" containerID="fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5" exitCode=0
Jul 10 20:49:43.839663 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.839627    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"
Jul 10 20:49:43.839861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.839661    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" event={"ID":"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20","Type":"ContainerDied","Data":"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"}
Jul 10 20:49:43.839861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.839714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5" event={"ID":"0e6cff9e-4b3b-4be7-bd6b-e561543e1b20","Type":"ContainerDied","Data":"381a8e186eac42cf793b35561d9fa2eb52089f39f01004ca588e7f50d4ea8e4e"}
Jul 10 20:49:43.839861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.839737    2564 scope.go:117] "RemoveContainer" containerID="fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"
Jul 10 20:49:43.848820 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.848799    2564 scope.go:117] "RemoveContainer" containerID="fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"
Jul 10 20:49:43.849082 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:49:43.849062    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5\": container with ID starting with fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5 not found: ID does not exist" containerID="fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"
Jul 10 20:49:43.849140 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.849094    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5"} err="failed to get container status \"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5\": rpc error: code = NotFound desc = could not find container \"fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5\": container with ID starting with fbdc4399dafa334956a9244e27e6e84a8df55f0116808a5fcef071364ff726f5 not found: ID does not exist"
Jul 10 20:49:43.861081 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.861059    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:43.866893 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:43.866871    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a405bf62-777486dcdf-2v7c5"]
Jul 10 20:49:44.784770 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:44.784738    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" path="/var/lib/kubelet/pods/0e6cff9e-4b3b-4be7-bd6b-e561543e1b20/volumes"
Jul 10 20:49:50.143971 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:49:50.143935    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:51:13.824391 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:13.824299    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:51:16.741121 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:16.741094    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:51:16.744870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:16.744848    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:51:23.329382 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:23.329340    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:51:29.167558 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.167527    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:29.167928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.167899    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" containerName="maas-api"
Jul 10 20:51:29.167928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.167913    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" containerName="maas-api"
Jul 10 20:51:29.167997 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.167972    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e6cff9e-4b3b-4be7-bd6b-e561543e1b20" containerName="maas-api"
Jul 10 20:51:29.170884 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.170866    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.173676 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.173654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3fe963c1-serving-cert\""
Jul 10 20:51:29.184255 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.184230    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:29.208125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.208095    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7g2\" (UniqueName: \"kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.208272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.208147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.208272 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.208171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.309136 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.309100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.309136 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.309138    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.309346 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.309204    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7g2\" (UniqueName: \"kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.309346 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:51:29.309251    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-serving-cert: secret "maas-api-e2e-mt-3fe963c1-serving-cert" not found
Jul 10 20:51:29.309346 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:51:29.309323    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls podName:7d75e9fa-c273-42b5-85d8-42a9f1daccc8 nodeName:}" failed. No retries permitted until 2026-07-10 20:51:29.809306966 +0000 UTC m=+1813.709612163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls") pod "maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" (UID: "7d75e9fa-c273-42b5-85d8-42a9f1daccc8") : secret "maas-api-e2e-mt-3fe963c1-serving-cert" not found
Jul 10 20:51:29.309938 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.309920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.320443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.320420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7g2\" (UniqueName: \"kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.812868 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.812834    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:29.815160 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:29.815136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") pod \"maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:30.081692 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:30.081596    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:30.216343 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:30.216309    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:30.219363 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:51:30.219338    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d75e9fa_c273_42b5_85d8_42a9f1daccc8.slice/crio-fbc2b08c8765c7a185d1a6ba23046a83aad3c4bf6efd3f42af7308875a1df068 WatchSource:0}: Error finding container fbc2b08c8765c7a185d1a6ba23046a83aad3c4bf6efd3f42af7308875a1df068: Status 404 returned error can't find the container with id fbc2b08c8765c7a185d1a6ba23046a83aad3c4bf6efd3f42af7308875a1df068
Jul 10 20:51:30.220701 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:30.220685    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:51:30.306834 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:30.306802    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" event={"ID":"7d75e9fa-c273-42b5-85d8-42a9f1daccc8","Type":"ContainerStarted","Data":"fbc2b08c8765c7a185d1a6ba23046a83aad3c4bf6efd3f42af7308875a1df068"}
Jul 10 20:51:31.313396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:31.313361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" event={"ID":"7d75e9fa-c273-42b5-85d8-42a9f1daccc8","Type":"ContainerStarted","Data":"9815b4532518fd742291a54a5bb4793584e657d815fc2761756717fad3ada033"}
Jul 10 20:51:31.313780 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:31.313505    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:31.331256 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:31.331211    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" podStartSLOduration=1.940257983 podStartE2EDuration="2.331195935s" totalImagesPullingTime="390.937952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:51:31.329717677 +0000 UTC m=+1815.230022912" watchObservedRunningTime="2026-07-10 20:51:31.331195935 +0000 UTC m=+1815.231501239"
Jul 10 20:51:34.424776 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:34.424729    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:51:37.322629 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:37.322594    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:41.146153 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.146114    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:41.146591 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.146448    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" podUID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" containerName="maas-api" containerID="cri-o://9815b4532518fd742291a54a5bb4793584e657d815fc2761756717fad3ada033" gracePeriod=30
Jul 10 20:51:41.361314 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.361279    2564 generic.go:356] "Generic (PLEG): container finished" podID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" containerID="9815b4532518fd742291a54a5bb4793584e657d815fc2761756717fad3ada033" exitCode=0
Jul 10 20:51:41.361475 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.361352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" event={"ID":"7d75e9fa-c273-42b5-85d8-42a9f1daccc8","Type":"ContainerDied","Data":"9815b4532518fd742291a54a5bb4793584e657d815fc2761756717fad3ada033"}
Jul 10 20:51:41.395484 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.395453    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:41.504700 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.504613    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h7g2\" (UniqueName: \"kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2\") pod \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") "
Jul 10 20:51:41.504700 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.504669    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") pod \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") "
Jul 10 20:51:41.504913 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.504775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca\") pod \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\" (UID: \"7d75e9fa-c273-42b5-85d8-42a9f1daccc8\") "
Jul 10 20:51:41.505166 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.505140    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d75e9fa-c273-42b5-85d8-42a9f1daccc8" (UID: "7d75e9fa-c273-42b5-85d8-42a9f1daccc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:51:41.506665 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.506643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d75e9fa-c273-42b5-85d8-42a9f1daccc8" (UID: "7d75e9fa-c273-42b5-85d8-42a9f1daccc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:51:41.506759 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.506723    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2" (OuterVolumeSpecName: "kube-api-access-8h7g2") pod "7d75e9fa-c273-42b5-85d8-42a9f1daccc8" (UID: "7d75e9fa-c273-42b5-85d8-42a9f1daccc8"). InnerVolumeSpecName "kube-api-access-8h7g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:51:41.605605 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.605562    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:51:41.605605 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.605599    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:51:41.605605 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:41.605611    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7g2\" (UniqueName: \"kubernetes.io/projected/7d75e9fa-c273-42b5-85d8-42a9f1daccc8-kube-api-access-8h7g2\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:51:42.366994 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.366960    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p" event={"ID":"7d75e9fa-c273-42b5-85d8-42a9f1daccc8","Type":"ContainerDied","Data":"fbc2b08c8765c7a185d1a6ba23046a83aad3c4bf6efd3f42af7308875a1df068"}
Jul 10 20:51:42.367424 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.367000    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"
Jul 10 20:51:42.367424 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.367024    2564 scope.go:117] "RemoveContainer" containerID="9815b4532518fd742291a54a5bb4793584e657d815fc2761756717fad3ada033"
Jul 10 20:51:42.389750 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.389725    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:42.391223 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.391203    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3fe963c1-6fcbf4d685-x7r5p"]
Jul 10 20:51:42.784667 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:42.784593    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" path="/var/lib/kubelet/pods/7d75e9fa-c273-42b5-85d8-42a9f1daccc8/volumes"
Jul 10 20:51:45.623403 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:51:45.623371    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:52:09.327128 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.327081    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:09.329435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.327631    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" containerName="maas-api"
Jul 10 20:52:09.329435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.327650    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" containerName="maas-api"
Jul 10 20:52:09.329435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.327762    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d75e9fa-c273-42b5-85d8-42a9f1daccc8" containerName="maas-api"
Jul 10 20:52:09.330839 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.330819    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.333371 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.333344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f59cd92e-serving-cert\""
Jul 10 20:52:09.342428 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.342404    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:09.425837 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.425807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.426017 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.425865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.426017 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.425948    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8wn7\" (UniqueName: \"kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.527266 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.527232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.527412 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.527288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.527412 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.527338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8wn7\" (UniqueName: \"kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.527513 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:52:09.527447    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-serving-cert: secret "maas-api-e2e-mt-f59cd92e-serving-cert" not found
Jul 10 20:52:09.527552 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:52:09.527540    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls podName:0987b03c-423a-445f-abb0-bd67582b84fd nodeName:}" failed. No retries permitted until 2026-07-10 20:52:10.027519961 +0000 UTC m=+1853.927825166 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls") pod "maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" (UID: "0987b03c-423a-445f-abb0-bd67582b84fd") : secret "maas-api-e2e-mt-f59cd92e-serving-cert" not found
Jul 10 20:52:09.527867 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.527849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:09.536605 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:09.536576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8wn7\" (UniqueName: \"kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:10.031824 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:10.031791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:10.034142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:10.034124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") pod \"maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:10.242032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:10.241997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:10.381876 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:10.381846    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:10.385401 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:52:10.385367    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0987b03c_423a_445f_abb0_bd67582b84fd.slice/crio-da4b355ec9588beb8ced47e151f250beedeb43a75ef88377a87192f6326e4ba6 WatchSource:0}: Error finding container da4b355ec9588beb8ced47e151f250beedeb43a75ef88377a87192f6326e4ba6: Status 404 returned error can't find the container with id da4b355ec9588beb8ced47e151f250beedeb43a75ef88377a87192f6326e4ba6
Jul 10 20:52:10.490074 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:10.490040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" event={"ID":"0987b03c-423a-445f-abb0-bd67582b84fd","Type":"ContainerStarted","Data":"da4b355ec9588beb8ced47e151f250beedeb43a75ef88377a87192f6326e4ba6"}
Jul 10 20:52:11.497770 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:11.497729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" event={"ID":"0987b03c-423a-445f-abb0-bd67582b84fd","Type":"ContainerStarted","Data":"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"}
Jul 10 20:52:11.498230 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:11.497848    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:11.517155 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:11.517111    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" podStartSLOduration=2.139742377 podStartE2EDuration="2.517095766s" totalImagesPullingTime="377.353389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:52:11.515971102 +0000 UTC m=+1855.416276322" watchObservedRunningTime="2026-07-10 20:52:11.517095766 +0000 UTC m=+1855.417400963"
Jul 10 20:52:14.121014 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:14.120980    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:52:17.508225 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:17.508199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:19.520067 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.520033    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:52:19.523433 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.523415    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.527765 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.527742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f59cd92e\"/\"default-dockercfg-sbldf\""
Jul 10 20:52:19.527897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.527803    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f59cd92e\"/\"kube-root-ca.crt\""
Jul 10 20:52:19.527897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.527819    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f59cd92e\"/\"openshift-service-ca.crt\""
Jul 10 20:52:19.528024 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.527742    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f59cd92e\"/\"e2e-lifecycle-model-f59cd92e-kserve-self-signed-certs\""
Jul 10 20:52:19.531986 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.531965    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:52:19.597896 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.597866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.598049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.597922    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.598049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.597949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknvx\" (UniqueName: \"kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.598049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.597997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.598049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.598034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.598217 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.598067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.698809 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.698775    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.698982 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.698905    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknvx\" (UniqueName: \"kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.698982 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.698973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699102 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699102 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699057    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699205 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699106    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699293 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699272    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699414 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.699557 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.699532    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.701513 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.701478    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.701748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.701731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.706092 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.706070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknvx\" (UniqueName: \"kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx\") pod \"e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") " pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.834474 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.834445    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:19.966204 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:19.966178    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:52:19.968011 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:52:19.967982    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7999f52a_2ae4_47d3_992b_660ecf59dee1.slice/crio-b3c65d04ba67a2afb2d5cfc92aca248dc42bc5fb1803abe373dccc77834b25c7 WatchSource:0}: Error finding container b3c65d04ba67a2afb2d5cfc92aca248dc42bc5fb1803abe373dccc77834b25c7: Status 404 returned error can't find the container with id b3c65d04ba67a2afb2d5cfc92aca248dc42bc5fb1803abe373dccc77834b25c7
Jul 10 20:52:20.540472 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:20.540437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" event={"ID":"7999f52a-2ae4-47d3-992b-660ecf59dee1","Type":"ContainerStarted","Data":"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"}
Jul 10 20:52:20.540871 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:20.540504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" event={"ID":"7999f52a-2ae4-47d3-992b-660ecf59dee1","Type":"ContainerStarted","Data":"b3c65d04ba67a2afb2d5cfc92aca248dc42bc5fb1803abe373dccc77834b25c7"}
Jul 10 20:52:20.559087 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:20.559036    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" podStartSLOduration=1.559022189 podStartE2EDuration="1.559022189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:52:20.558419565 +0000 UTC m=+1864.458724786" watchObservedRunningTime="2026-07-10 20:52:20.559022189 +0000 UTC m=+1864.459327449"
Jul 10 20:52:23.024703 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:23.024667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:52:29.835233 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:29.835199    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:29.835233 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:29.835243    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:29.847623 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:29.847602    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:30.601670 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:30.601634    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:52:38.520521 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:38.520438    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:52:41.253131 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.253099    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:41.253585 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.253346    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" podUID="0987b03c-423a-445f-abb0-bd67582b84fd" containerName="maas-api" containerID="cri-o://6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a" gracePeriod=30
Jul 10 20:52:41.503889 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.503838    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:41.555581 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.555552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8wn7\" (UniqueName: \"kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7\") pod \"0987b03c-423a-445f-abb0-bd67582b84fd\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") "
Jul 10 20:52:41.555714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.555632    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca\") pod \"0987b03c-423a-445f-abb0-bd67582b84fd\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") "
Jul 10 20:52:41.555714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.555670    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") pod \"0987b03c-423a-445f-abb0-bd67582b84fd\" (UID: \"0987b03c-423a-445f-abb0-bd67582b84fd\") "
Jul 10 20:52:41.555980 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.555957    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0987b03c-423a-445f-abb0-bd67582b84fd" (UID: "0987b03c-423a-445f-abb0-bd67582b84fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:52:41.557633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.557610    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0987b03c-423a-445f-abb0-bd67582b84fd" (UID: "0987b03c-423a-445f-abb0-bd67582b84fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:52:41.557973 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.557953    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7" (OuterVolumeSpecName: "kube-api-access-t8wn7") pod "0987b03c-423a-445f-abb0-bd67582b84fd" (UID: "0987b03c-423a-445f-abb0-bd67582b84fd"). InnerVolumeSpecName "kube-api-access-t8wn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:52:41.637522 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.637481    2564 generic.go:356] "Generic (PLEG): container finished" podID="0987b03c-423a-445f-abb0-bd67582b84fd" containerID="6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a" exitCode=0
Jul 10 20:52:41.637633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.637539    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"
Jul 10 20:52:41.637633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.637581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" event={"ID":"0987b03c-423a-445f-abb0-bd67582b84fd","Type":"ContainerDied","Data":"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"}
Jul 10 20:52:41.637633 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.637631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn" event={"ID":"0987b03c-423a-445f-abb0-bd67582b84fd","Type":"ContainerDied","Data":"da4b355ec9588beb8ced47e151f250beedeb43a75ef88377a87192f6326e4ba6"}
Jul 10 20:52:41.637775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.637654    2564 scope.go:117] "RemoveContainer" containerID="6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"
Jul 10 20:52:41.646367 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.646351    2564 scope.go:117] "RemoveContainer" containerID="6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"
Jul 10 20:52:41.646647 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:52:41.646627    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a\": container with ID starting with 6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a not found: ID does not exist" containerID="6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"
Jul 10 20:52:41.646715 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.646659    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a"} err="failed to get container status \"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a\": rpc error: code = NotFound desc = could not find container \"6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a\": container with ID starting with 6fbe43ddd9818f2d8904d51996a1812aba712bee14dd2097f7038a17570c7c4a not found: ID does not exist"
Jul 10 20:52:41.657076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.657056    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0987b03c-423a-445f-abb0-bd67582b84fd-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:52:41.657183 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.657081    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0987b03c-423a-445f-abb0-bd67582b84fd-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:52:41.657183 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.657095    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8wn7\" (UniqueName: \"kubernetes.io/projected/0987b03c-423a-445f-abb0-bd67582b84fd-kube-api-access-t8wn7\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:52:41.659996 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.659973    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:41.661526 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:41.661503    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f59cd92e-7fc84f59d-8q2bn"]
Jul 10 20:52:42.784530 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:42.784476    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0987b03c-423a-445f-abb0-bd67582b84fd" path="/var/lib/kubelet/pods/0987b03c-423a-445f-abb0-bd67582b84fd/volumes"
Jul 10 20:52:49.325564 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:49.325531    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:52:51.944644 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:52:51.944611    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:53:06.964630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:06.964569    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" podUID="7999f52a-2ae4-47d3-992b-660ecf59dee1" containerName="main" containerID="cri-o://c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35" gracePeriod=45
Jul 10 20:53:07.218269 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.218214    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:53:07.349851 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.349815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.349851 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.349859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.350066 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.349891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.350066 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.349934    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.350066 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.349984    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.350066 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.350016    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gknvx\" (UniqueName: \"kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx\") pod \"7999f52a-2ae4-47d3-992b-660ecf59dee1\" (UID: \"7999f52a-2ae4-47d3-992b-660ecf59dee1\") "
Jul 10 20:53:07.350289 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.350200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:53:07.350289 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.350231    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:53:07.350289 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.350239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home" (OuterVolumeSpecName: "home") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:53:07.351970 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.351945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm" (OuterVolumeSpecName: "dshm") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:53:07.352072 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.351946    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:53:07.352072 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.352052    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx" (OuterVolumeSpecName: "kube-api-access-gknvx") pod "7999f52a-2ae4-47d3-992b-660ecf59dee1" (UID: "7999f52a-2ae4-47d3-992b-660ecf59dee1"). InnerVolumeSpecName "kube-api-access-gknvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:53:07.451059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451036    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.451059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451057    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gknvx\" (UniqueName: \"kubernetes.io/projected/7999f52a-2ae4-47d3-992b-660ecf59dee1-kube-api-access-gknvx\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.451193 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451067    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7999f52a-2ae4-47d3-992b-660ecf59dee1-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.451193 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451075    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.451193 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451083    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.451193 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.451090    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7999f52a-2ae4-47d3-992b-660ecf59dee1-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:53:07.752466 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.752442    2564 generic.go:356] "Generic (PLEG): container finished" podID="7999f52a-2ae4-47d3-992b-660ecf59dee1" containerID="c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35" exitCode=0
Jul 10 20:53:07.752589 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.752481    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"
Jul 10 20:53:07.752589 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.752526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" event={"ID":"7999f52a-2ae4-47d3-992b-660ecf59dee1","Type":"ContainerDied","Data":"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"}
Jul 10 20:53:07.752589 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.752572    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9" event={"ID":"7999f52a-2ae4-47d3-992b-660ecf59dee1","Type":"ContainerDied","Data":"b3c65d04ba67a2afb2d5cfc92aca248dc42bc5fb1803abe373dccc77834b25c7"}
Jul 10 20:53:07.752693 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.752594    2564 scope.go:117] "RemoveContainer" containerID="c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"
Jul 10 20:53:07.761585 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.761567    2564 scope.go:117] "RemoveContainer" containerID="c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"
Jul 10 20:53:07.761831 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:53:07.761815    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35\": container with ID starting with c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35 not found: ID does not exist" containerID="c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"
Jul 10 20:53:07.761874 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.761840    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35"} err="failed to get container status \"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35\": rpc error: code = NotFound desc = could not find container \"c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35\": container with ID starting with c49c34ae75a5e45222dd0b19c2debbd518271020319f0be6fa9ce634fd471f35 not found: ID does not exist"
Jul 10 20:53:07.773429 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.773409    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:53:07.777841 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:07.777817    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f59cd92e/e2e-lifecycle-model-f59cd92e-kserve-64999dd44c-q7tt9"]
Jul 10 20:53:08.784284 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:08.784254    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7999f52a-2ae4-47d3-992b-660ecf59dee1" path="/var/lib/kubelet/pods/7999f52a-2ae4-47d3-992b-660ecf59dee1/volumes"
Jul 10 20:53:31.728952 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:31.728915    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:53:41.273551 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:41.273516    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:53:52.526030 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:53:52.525994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:54:32.058603 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.058523    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.058878    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7999f52a-2ae4-47d3-992b-660ecf59dee1" containerName="main"
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.058890    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7999f52a-2ae4-47d3-992b-660ecf59dee1" containerName="main"
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.058914    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0987b03c-423a-445f-abb0-bd67582b84fd" containerName="maas-api"
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.058920    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0987b03c-423a-445f-abb0-bd67582b84fd" containerName="maas-api"
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.059000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0987b03c-423a-445f-abb0-bd67582b84fd" containerName="maas-api"
Jul 10 20:54:32.059049 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.059010    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7999f52a-2ae4-47d3-992b-660ecf59dee1" containerName="main"
Jul 10 20:54:32.061949 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.061926    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.065112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.065084    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-de7153-serving-cert\""
Jul 10 20:54:32.075185 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.075159    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:54:32.210815 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.210776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.211027 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.210854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.211027 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.210917    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4757p\" (UniqueName: \"kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.311670 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.311557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.311670 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.311634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.311670 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.311663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4757p\" (UniqueName: \"kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.312299 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.312275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.314177 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.314138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.320897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.320860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4757p\" (UniqueName: \"kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p\") pod \"maas-api-e2e-inf-a-de7153-988c84f4d-j8f64\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.373461 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.373402    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:32.508971 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:32.508941    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:54:32.511823 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:54:32.511781    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b56f7_4906_471c_9ab5_9a1094d8d3b3.slice/crio-dec49107f6d62db53f41d01b408f4084d559848d53a9ae0c08650e8bc82d104a WatchSource:0}: Error finding container dec49107f6d62db53f41d01b408f4084d559848d53a9ae0c08650e8bc82d104a: Status 404 returned error can't find the container with id dec49107f6d62db53f41d01b408f4084d559848d53a9ae0c08650e8bc82d104a
Jul 10 20:54:33.131638 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:33.131600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" event={"ID":"e86b56f7-4906-471c-9ab5-9a1094d8d3b3","Type":"ContainerStarted","Data":"dec49107f6d62db53f41d01b408f4084d559848d53a9ae0c08650e8bc82d104a"}
Jul 10 20:54:34.138848 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:34.138810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" event={"ID":"e86b56f7-4906-471c-9ab5-9a1094d8d3b3","Type":"ContainerStarted","Data":"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"}
Jul 10 20:54:34.139234 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:34.138911    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:34.156625 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:34.156580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" podStartSLOduration=0.947432404 podStartE2EDuration="2.156565951s" totalImagesPullingTime="1.209133547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:54:34.154425864 +0000 UTC m=+1998.054731084" watchObservedRunningTime="2026-07-10 20:54:34.156565951 +0000 UTC m=+1998.056871169"
Jul 10 20:54:39.755444 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:39.755409    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:54:40.093045 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.093011    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:54:40.096474 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.096455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.099457 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.099434    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0b0cab-serving-cert\""
Jul 10 20:54:40.106645 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.106625    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:54:40.148834 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.148793    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:54:40.277548 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.277483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.277765 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.277738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsbc\" (UniqueName: \"kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.277843 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.277817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.379223 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.379117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.379403 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.379282    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsbc\" (UniqueName: \"kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.379403 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.379329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.379898 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.379874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.381747 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.381729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.387658 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.387629    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsbc\" (UniqueName: \"kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc\") pod \"maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.409303 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.409278    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:40.543176 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:40.543133    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:54:40.544896 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:54:40.544861    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb03d22a4_585c_4da8_a328_f200a9ea6c61.slice/crio-3c15d426b7606d94257133eab8a385ad46e6a201809b005bec2061ec7d3b2317 WatchSource:0}: Error finding container 3c15d426b7606d94257133eab8a385ad46e6a201809b005bec2061ec7d3b2317: Status 404 returned error can't find the container with id 3c15d426b7606d94257133eab8a385ad46e6a201809b005bec2061ec7d3b2317
Jul 10 20:54:41.173941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:41.173843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" event={"ID":"b03d22a4-585c-4da8-a328-f200a9ea6c61","Type":"ContainerStarted","Data":"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"}
Jul 10 20:54:41.173941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:41.173894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" event={"ID":"b03d22a4-585c-4da8-a328-f200a9ea6c61","Type":"ContainerStarted","Data":"3c15d426b7606d94257133eab8a385ad46e6a201809b005bec2061ec7d3b2317"}
Jul 10 20:54:41.173941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:41.173927    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:41.194173 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:41.194122    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" podStartSLOduration=0.824897044 podStartE2EDuration="1.194104991s" totalImagesPullingTime="369.207947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:54:41.192426354 +0000 UTC m=+2005.092731594" watchObservedRunningTime="2026-07-10 20:54:41.194104991 +0000 UTC m=+2005.094410272"
Jul 10 20:54:47.184094 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:47.184060    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:54:51.720810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:51.720768    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:54:57.821260 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.821222    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:54:57.824917 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.824889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.829269 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.829250    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-de7153\"/\"kube-root-ca.crt\""
Jul 10 20:54:57.829417 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.829268    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-de7153\"/\"default-dockercfg-qhsv6\""
Jul 10 20:54:57.829417 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.829291    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-de7153\"/\"openshift-service-ca.crt\""
Jul 10 20:54:57.829417 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.829248    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-de7153\"/\"test-model-de7153-kserve-self-signed-certs\""
Jul 10 20:54:57.832928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.832911    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:54:57.898892 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.898865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.899032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.898896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.899032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.898967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.899032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.898984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2cc\" (UniqueName: \"kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.899032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.899011    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:57.899032 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.899027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000011 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:57.999984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2cc\" (UniqueName: \"kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000041    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000064    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000276 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000249    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000341 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000438 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000421    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000520 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.000642 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.000623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.002336 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.002317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.002504 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.002477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.008965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.008943    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2cc\" (UniqueName: \"kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc\") pod \"test-model-de7153-kserve-5b4fbcdff-g4xsq\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") " pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.135343 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.135284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:54:58.264206 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:58.264144    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:54:58.266335 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:54:58.266308    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52046923_6c7b_4033_bee5_fc45bfd8eef7.slice/crio-e0884ff4b1bbe965a8c87d16a7539157403c30dfa823c9bdf27a2536d58ac89d WatchSource:0}: Error finding container e0884ff4b1bbe965a8c87d16a7539157403c30dfa823c9bdf27a2536d58ac89d: Status 404 returned error can't find the container with id e0884ff4b1bbe965a8c87d16a7539157403c30dfa823c9bdf27a2536d58ac89d
Jul 10 20:54:59.257905 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:59.257865    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" event={"ID":"52046923-6c7b-4033-bee5-fc45bfd8eef7","Type":"ContainerStarted","Data":"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"}
Jul 10 20:54:59.257905 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:59.257914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" event={"ID":"52046923-6c7b-4033-bee5-fc45bfd8eef7","Type":"ContainerStarted","Data":"e0884ff4b1bbe965a8c87d16a7539157403c30dfa823c9bdf27a2536d58ac89d"}
Jul 10 20:54:59.275523 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:54:59.275454    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" podStartSLOduration=2.2754395 podStartE2EDuration="2.2754395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:54:59.275048229 +0000 UTC m=+2023.175353473" watchObservedRunningTime="2026-07-10 20:54:59.2754395 +0000 UTC m=+2023.175744734"
Jul 10 20:55:04.453556 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:04.453516    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:55:08.135468 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:08.135430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:08.135868 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:08.135484    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:08.147866 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:08.147844    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:08.310766 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:08.310742    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:20.921793 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.921762    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:55:20.925059 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.925041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.929722 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.929690    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b0cab\"/\"kube-root-ca.crt\""
Jul 10 20:55:20.929722 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.929703    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0b0cab\"/\"openshift-service-ca.crt\""
Jul 10 20:55:20.929908 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.929756    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b0cab\"/\"test-model-0b0cab-kserve-self-signed-certs\""
Jul 10 20:55:20.929908 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.929762    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0b0cab\"/\"default-dockercfg-wpbh5\""
Jul 10 20:55:20.934541 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.934518    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:55:20.983965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.983932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.984083 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.983970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tsln\" (UniqueName: \"kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.984083 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.984045    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.984155 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.984103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.984194 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.984156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:20.984231 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:20.984213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085203 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tsln\" (UniqueName: \"kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085359 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085314    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085668 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085562    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085668 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085621    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.085775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.085724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.087517 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.087482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.087713 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.087695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.096887 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.096863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tsln\" (UniqueName: \"kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln\") pod \"test-model-0b0cab-kserve-869498cb9-jkpg7\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") " pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.234836 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.234767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:21.366159 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:21.366126    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:55:21.368276 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:55:21.368251    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3966ee3f_fdce_490d_95cd_cc6d23b662c2.slice/crio-55219a531f51f78167734468c72dca13dc88023fbba49bbc32faa7cdb419b502 WatchSource:0}: Error finding container 55219a531f51f78167734468c72dca13dc88023fbba49bbc32faa7cdb419b502: Status 404 returned error can't find the container with id 55219a531f51f78167734468c72dca13dc88023fbba49bbc32faa7cdb419b502
Jul 10 20:55:22.362531 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:22.362480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" event={"ID":"3966ee3f-fdce-490d-95cd-cc6d23b662c2","Type":"ContainerStarted","Data":"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"}
Jul 10 20:55:22.362908 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:22.362543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" event={"ID":"3966ee3f-fdce-490d-95cd-cc6d23b662c2","Type":"ContainerStarted","Data":"55219a531f51f78167734468c72dca13dc88023fbba49bbc32faa7cdb419b502"}
Jul 10 20:55:22.380979 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:22.380928    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" podStartSLOduration=2.380912636 podStartE2EDuration="2.380912636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:55:22.38055917 +0000 UTC m=+2046.280864408" watchObservedRunningTime="2026-07-10 20:55:22.380912636 +0000 UTC m=+2046.281217855"
Jul 10 20:55:24.925456 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:24.925424    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:55:31.235280 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:31.235247    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:31.235673 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:31.235294    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:31.247864 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:31.247840    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:31.414381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:31.414346    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:43.015915 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:43.015838    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:55:43.800340 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:43.800309    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:55:44.696601 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:44.696570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:55:44.697074 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:44.696796    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" podUID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" containerName="maas-api" containerID="cri-o://77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8" gracePeriod=30
Jul 10 20:55:44.948047 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:44.947990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:55:45.091203 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.091178    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca\") pod \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") "
Jul 10 20:55:45.091320 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.091209    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4757p\" (UniqueName: \"kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p\") pod \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") "
Jul 10 20:55:45.091320 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.091272    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls\") pod \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\" (UID: \"e86b56f7-4906-471c-9ab5-9a1094d8d3b3\") "
Jul 10 20:55:45.091602 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.091578    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e86b56f7-4906-471c-9ab5-9a1094d8d3b3" (UID: "e86b56f7-4906-471c-9ab5-9a1094d8d3b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:55:45.093270 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.093246    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p" (OuterVolumeSpecName: "kube-api-access-4757p") pod "e86b56f7-4906-471c-9ab5-9a1094d8d3b3" (UID: "e86b56f7-4906-471c-9ab5-9a1094d8d3b3"). InnerVolumeSpecName "kube-api-access-4757p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:55:45.093357 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.093329    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e86b56f7-4906-471c-9ab5-9a1094d8d3b3" (UID: "e86b56f7-4906-471c-9ab5-9a1094d8d3b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:55:45.192408 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.192384    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:45.192408 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.192407    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:45.192563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.192416    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4757p\" (UniqueName: \"kubernetes.io/projected/e86b56f7-4906-471c-9ab5-9a1094d8d3b3-kube-api-access-4757p\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:45.469379 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.469347    2564 generic.go:356] "Generic (PLEG): container finished" podID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" containerID="77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8" exitCode=0
Jul 10 20:55:45.469563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.469396    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"
Jul 10 20:55:45.469563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.469429    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" event={"ID":"e86b56f7-4906-471c-9ab5-9a1094d8d3b3","Type":"ContainerDied","Data":"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"}
Jul 10 20:55:45.469563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.469483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64" event={"ID":"e86b56f7-4906-471c-9ab5-9a1094d8d3b3","Type":"ContainerDied","Data":"dec49107f6d62db53f41d01b408f4084d559848d53a9ae0c08650e8bc82d104a"}
Jul 10 20:55:45.469563 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.469523    2564 scope.go:117] "RemoveContainer" containerID="77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"
Jul 10 20:55:45.478144 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.478127    2564 scope.go:117] "RemoveContainer" containerID="77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"
Jul 10 20:55:45.478428 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:55:45.478411    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8\": container with ID starting with 77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8 not found: ID does not exist" containerID="77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"
Jul 10 20:55:45.478507 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.478435    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8"} err="failed to get container status \"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8\": rpc error: code = NotFound desc = could not find container \"77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8\": container with ID starting with 77b9b1ee558593977cd80957ff882d4ce55db84984ebb990db9da968305187d8 not found: ID does not exist"
Jul 10 20:55:45.490943 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.490922    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:55:45.496972 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:45.496949    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-de7153-988c84f4d-j8f64"]
Jul 10 20:55:46.783867 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:46.783835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" path="/var/lib/kubelet/pods/e86b56f7-4906-471c-9ab5-9a1094d8d3b3/volumes"
Jul 10 20:55:49.329660 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:49.329618    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:55:58.035775 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.035718    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" podUID="52046923-6c7b-4033-bee5-fc45bfd8eef7" containerName="main" containerID="cri-o://7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b" gracePeriod=45
Jul 10 20:55:58.287982 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.287925    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:58.381339 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381312    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381347    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381381    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm2cc\" (UniqueName: \"kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381441    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381666 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381480    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm\") pod \"52046923-6c7b-4033-bee5-fc45bfd8eef7\" (UID: \"52046923-6c7b-4033-bee5-fc45bfd8eef7\") "
Jul 10 20:55:58.381666 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:58.381769 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381660    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home" (OuterVolumeSpecName: "home") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:58.381821 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381763    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache" (OuterVolumeSpecName: "model-cache") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:58.381870 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381849    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.381921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381868    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.381921 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.381881    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.383595 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.383566    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:55:58.383695 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.383615    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm" (OuterVolumeSpecName: "dshm") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:58.383695 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.383630    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc" (OuterVolumeSpecName: "kube-api-access-pm2cc") pod "52046923-6c7b-4033-bee5-fc45bfd8eef7" (UID: "52046923-6c7b-4033-bee5-fc45bfd8eef7"). InnerVolumeSpecName "kube-api-access-pm2cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:55:58.483721 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.483696    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52046923-6c7b-4033-bee5-fc45bfd8eef7-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.483827 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.483734    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2cc\" (UniqueName: \"kubernetes.io/projected/52046923-6c7b-4033-bee5-fc45bfd8eef7-kube-api-access-pm2cc\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.483827 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.483745    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52046923-6c7b-4033-bee5-fc45bfd8eef7-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:58.531445 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.531419    2564 generic.go:356] "Generic (PLEG): container finished" podID="52046923-6c7b-4033-bee5-fc45bfd8eef7" containerID="7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b" exitCode=0
Jul 10 20:55:58.531630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.531473    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"
Jul 10 20:55:58.531630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.531519    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" event={"ID":"52046923-6c7b-4033-bee5-fc45bfd8eef7","Type":"ContainerDied","Data":"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"}
Jul 10 20:55:58.531630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.531565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq" event={"ID":"52046923-6c7b-4033-bee5-fc45bfd8eef7","Type":"ContainerDied","Data":"e0884ff4b1bbe965a8c87d16a7539157403c30dfa823c9bdf27a2536d58ac89d"}
Jul 10 20:55:58.531630 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.531587    2564 scope.go:117] "RemoveContainer" containerID="7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"
Jul 10 20:55:58.540244 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.540230    2564 scope.go:117] "RemoveContainer" containerID="7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"
Jul 10 20:55:58.540541 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:55:58.540518    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b\": container with ID starting with 7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b not found: ID does not exist" containerID="7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"
Jul 10 20:55:58.540641 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.540547    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b"} err="failed to get container status \"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b\": rpc error: code = NotFound desc = could not find container \"7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b\": container with ID starting with 7b0a5001d95edc9783f1b60124db095b0dbf5df70d530286c05f496af6468f9b not found: ID does not exist"
Jul 10 20:55:58.552793 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.552772    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:55:58.556164 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.556145    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-de7153/test-model-de7153-kserve-5b4fbcdff-g4xsq"]
Jul 10 20:55:58.785502 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.785467    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52046923-6c7b-4033-bee5-fc45bfd8eef7" path="/var/lib/kubelet/pods/52046923-6c7b-4033-bee5-fc45bfd8eef7/volumes"
Jul 10 20:55:58.820198 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:58.820151    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" podUID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" containerName="main" containerID="cri-o://65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d" gracePeriod=45
Jul 10 20:55:59.070252 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.070202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:59.089094 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089070    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089202 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089117    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tsln\" (UniqueName: \"kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089202 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089147    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089202 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089170    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089202 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089195    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089405 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089223    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm\") pod \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\" (UID: \"3966ee3f-fdce-490d-95cd-cc6d23b662c2\") "
Jul 10 20:55:59.089535 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089509    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home" (OuterVolumeSpecName: "home") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:59.089657 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089634    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:59.089723 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.089704    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:59.091382 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.091356    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:55:59.091548 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.091467    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln" (OuterVolumeSpecName: "kube-api-access-9tsln") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "kube-api-access-9tsln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:55:59.091737 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.091710    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm" (OuterVolumeSpecName: "dshm") pod "3966ee3f-fdce-490d-95cd-cc6d23b662c2" (UID: "3966ee3f-fdce-490d-95cd-cc6d23b662c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 20:55:59.189806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189784    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.189806 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189805    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.189941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189819    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.189941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189830    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tsln\" (UniqueName: \"kubernetes.io/projected/3966ee3f-fdce-490d-95cd-cc6d23b662c2-kube-api-access-9tsln\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.189941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189841    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.189941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.189853    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3966ee3f-fdce-490d-95cd-cc6d23b662c2-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:55:59.539570 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.539476    2564 generic.go:356] "Generic (PLEG): container finished" podID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" containerID="65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d" exitCode=0
Jul 10 20:55:59.539570 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.539537    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"
Jul 10 20:55:59.539752 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.539565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" event={"ID":"3966ee3f-fdce-490d-95cd-cc6d23b662c2","Type":"ContainerDied","Data":"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"}
Jul 10 20:55:59.539752 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.539617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7" event={"ID":"3966ee3f-fdce-490d-95cd-cc6d23b662c2","Type":"ContainerDied","Data":"55219a531f51f78167734468c72dca13dc88023fbba49bbc32faa7cdb419b502"}
Jul 10 20:55:59.539752 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.539640    2564 scope.go:117] "RemoveContainer" containerID="65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"
Jul 10 20:55:59.548161 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.548144    2564 scope.go:117] "RemoveContainer" containerID="65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"
Jul 10 20:55:59.548382 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:55:59.548363    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d\": container with ID starting with 65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d not found: ID does not exist" containerID="65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"
Jul 10 20:55:59.548435 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.548390    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d"} err="failed to get container status \"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d\": rpc error: code = NotFound desc = could not find container \"65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d\": container with ID starting with 65e755a40719c052a7b69c8a53d793eff01cad6b5e01220b94c0efd0696d778d not found: ID does not exist"
Jul 10 20:55:59.560819 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.560799    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:55:59.564283 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:55:59.564259    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0b0cab/test-model-0b0cab-kserve-869498cb9-jkpg7"]
Jul 10 20:56:00.784076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:00.784042    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" path="/var/lib/kubelet/pods/3966ee3f-fdce-490d-95cd-cc6d23b662c2/volumes"
Jul 10 20:56:16.766456 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:16.766429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:56:16.771275 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:16.771256    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 20:56:27.936749 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:27.936715    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:56:27.937207 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:27.936936    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" podUID="b03d22a4-585c-4da8-a328-f200a9ea6c61" containerName="maas-api" containerID="cri-o://42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b" gracePeriod=30
Jul 10 20:56:29.185908 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.185882    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:56:29.318167 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.318134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npsbc\" (UniqueName: \"kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc\") pod \"b03d22a4-585c-4da8-a328-f200a9ea6c61\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") "
Jul 10 20:56:29.318360 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.318188    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca\") pod \"b03d22a4-585c-4da8-a328-f200a9ea6c61\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") "
Jul 10 20:56:29.318360 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.318263    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls\") pod \"b03d22a4-585c-4da8-a328-f200a9ea6c61\" (UID: \"b03d22a4-585c-4da8-a328-f200a9ea6c61\") "
Jul 10 20:56:29.318624 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.318596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b03d22a4-585c-4da8-a328-f200a9ea6c61" (UID: "b03d22a4-585c-4da8-a328-f200a9ea6c61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:56:29.320379 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.320353    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc" (OuterVolumeSpecName: "kube-api-access-npsbc") pod "b03d22a4-585c-4da8-a328-f200a9ea6c61" (UID: "b03d22a4-585c-4da8-a328-f200a9ea6c61"). InnerVolumeSpecName "kube-api-access-npsbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:56:29.320470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.320373    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b03d22a4-585c-4da8-a328-f200a9ea6c61" (UID: "b03d22a4-585c-4da8-a328-f200a9ea6c61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:56:29.418880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.418838    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsbc\" (UniqueName: \"kubernetes.io/projected/b03d22a4-585c-4da8-a328-f200a9ea6c61-kube-api-access-npsbc\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:29.418880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.418872    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b03d22a4-585c-4da8-a328-f200a9ea6c61-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:29.418880 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.418884    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b03d22a4-585c-4da8-a328-f200a9ea6c61-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:29.678562 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.678453    2564 generic.go:356] "Generic (PLEG): container finished" podID="b03d22a4-585c-4da8-a328-f200a9ea6c61" containerID="42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b" exitCode=0
Jul 10 20:56:29.678562 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.678513    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"
Jul 10 20:56:29.678562 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.678540    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" event={"ID":"b03d22a4-585c-4da8-a328-f200a9ea6c61","Type":"ContainerDied","Data":"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"}
Jul 10 20:56:29.678845 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.678627    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm" event={"ID":"b03d22a4-585c-4da8-a328-f200a9ea6c61","Type":"ContainerDied","Data":"3c15d426b7606d94257133eab8a385ad46e6a201809b005bec2061ec7d3b2317"}
Jul 10 20:56:29.678845 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.678650    2564 scope.go:117] "RemoveContainer" containerID="42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"
Jul 10 20:56:29.687248 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.687229    2564 scope.go:117] "RemoveContainer" containerID="42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"
Jul 10 20:56:29.687519 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:29.687481    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b\": container with ID starting with 42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b not found: ID does not exist" containerID="42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"
Jul 10 20:56:29.687574 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.687529    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b"} err="failed to get container status \"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b\": rpc error: code = NotFound desc = could not find container \"42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b\": container with ID starting with 42ec74bffa91804eb27c7a425a2413c13d360d060abb5eee09cc3f2a7145db7b not found: ID does not exist"
Jul 10 20:56:29.700376 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.700350    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:56:29.703311 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:29.703288    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0b0cab-7f55cb88c6-vc5dm"]
Jul 10 20:56:30.784691 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:30.784655    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03d22a4-585c-4da8-a328-f200a9ea6c61" path="/var/lib/kubelet/pods/b03d22a4-585c-4da8-a328-f200a9ea6c61/volumes"
Jul 10 20:56:30.819522 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:30.819472    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:56:37.758398 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758366    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758702    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" containerName="maas-api"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758713    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" containerName="maas-api"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758725    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52046923-6c7b-4033-bee5-fc45bfd8eef7" containerName="main"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758730    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="52046923-6c7b-4033-bee5-fc45bfd8eef7" containerName="main"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758743    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03d22a4-585c-4da8-a328-f200a9ea6c61" containerName="maas-api"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758749    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03d22a4-585c-4da8-a328-f200a9ea6c61" containerName="maas-api"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758759    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" containerName="main"
Jul 10 20:56:37.758808 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758764    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" containerName="main"
Jul 10 20:56:37.759137 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758817    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="52046923-6c7b-4033-bee5-fc45bfd8eef7" containerName="main"
Jul 10 20:56:37.759137 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758825    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3966ee3f-fdce-490d-95cd-cc6d23b662c2" containerName="main"
Jul 10 20:56:37.759137 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758831    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03d22a4-585c-4da8-a328-f200a9ea6c61" containerName="maas-api"
Jul 10 20:56:37.759137 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.758840    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e86b56f7-4906-471c-9ab5-9a1094d8d3b3" containerName="maas-api"
Jul 10 20:56:37.763058 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.763036    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.766037 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.766015    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ec0f24-serving-cert\""
Jul 10 20:56:37.772299 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.772276    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:56:37.883840 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.883804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.884024 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.883869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22b4\" (UniqueName: \"kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.884024 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.883932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.985164 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.985123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.985361 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.985218    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22b4\" (UniqueName: \"kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.985361 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.985252    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.985519 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:37.985373    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-serving-cert: secret "maas-api-e2e-api-a-ec0f24-serving-cert" not found
Jul 10 20:56:37.985519 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:37.985445    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls podName:c3f4ae34-bfc9-4546-850a-0df1ed45ddd6 nodeName:}" failed. No retries permitted until 2026-07-10 20:56:38.485422229 +0000 UTC m=+2122.385727444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls") pod "maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" (UID: "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6") : secret "maas-api-e2e-api-a-ec0f24-serving-cert" not found
Jul 10 20:56:37.985906 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.985884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:37.995218 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:37.995187    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22b4\" (UniqueName: \"kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:38.489982 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:38.489946    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:38.492484 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:38.492449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") pod \"maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:38.673378 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:38.673342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:38.811402 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:38.811379    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:56:38.813445 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:56:38.813417    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f4ae34_bfc9_4546_850a_0df1ed45ddd6.slice/crio-31f130ade1da3d830a3cf4697ecc82bddb1ddc7eaca71cdc836569cb32359ac3 WatchSource:0}: Error finding container 31f130ade1da3d830a3cf4697ecc82bddb1ddc7eaca71cdc836569cb32359ac3: Status 404 returned error can't find the container with id 31f130ade1da3d830a3cf4697ecc82bddb1ddc7eaca71cdc836569cb32359ac3
Jul 10 20:56:38.814774 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:38.814755    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 10 20:56:39.728019 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:39.727980    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" event={"ID":"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6","Type":"ContainerStarted","Data":"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"}
Jul 10 20:56:39.728203 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:39.728040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" event={"ID":"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6","Type":"ContainerStarted","Data":"31f130ade1da3d830a3cf4697ecc82bddb1ddc7eaca71cdc836569cb32359ac3"}
Jul 10 20:56:39.728203 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:39.728077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:39.749665 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:39.749616    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" podStartSLOduration=2.354530581 podStartE2EDuration="2.749599199s" totalImagesPullingTime="395.068618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:56:39.748607541 +0000 UTC m=+2123.648912769" watchObservedRunningTime="2026-07-10 20:56:39.749599199 +0000 UTC m=+2123.649904419"
Jul 10 20:56:45.430504 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.430455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:56:45.737539 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.737446    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:45.928746 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.928713    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:56:45.932116 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.932095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:45.934992 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.934923    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-00c974-serving-cert\""
Jul 10 20:56:45.942857 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:45.942834    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:56:46.058396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.058309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.058396 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.058364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttjg\" (UniqueName: \"kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.058601 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.058438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.159640 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.159604    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.159786 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.159660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttjg\" (UniqueName: \"kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.159786 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.159696    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.159863 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:46.159810    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-serving-cert: secret "maas-api-e2e-api-b-00c974-serving-cert" not found
Jul 10 20:56:46.159906 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:46.159866    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls podName:cae766ad-86df-4df0-b3cf-835a3ae4a55b nodeName:}" failed. No retries permitted until 2026-07-10 20:56:46.659848034 +0000 UTC m=+2130.560153231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls") pod "maas-api-e2e-api-b-00c974-66658bb86b-wgns4" (UID: "cae766ad-86df-4df0-b3cf-835a3ae4a55b") : secret "maas-api-e2e-api-b-00c974-serving-cert" not found
Jul 10 20:56:46.160242 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.160222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.168388 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.168364    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttjg\" (UniqueName: \"kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.664332 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.664302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.666783 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.666757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") pod \"maas-api-e2e-api-b-00c974-66658bb86b-wgns4\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.844100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.844068    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:46.981047 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:46.981019    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:56:46.983310 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:56:46.983280    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae766ad_86df_4df0_b3cf_835a3ae4a55b.slice/crio-977b4eb19d604fdf5d8316e539372504a201d3afbe20d06b381b72b037588c06 WatchSource:0}: Error finding container 977b4eb19d604fdf5d8316e539372504a201d3afbe20d06b381b72b037588c06: Status 404 returned error can't find the container with id 977b4eb19d604fdf5d8316e539372504a201d3afbe20d06b381b72b037588c06
Jul 10 20:56:47.770217 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:47.770130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" event={"ID":"cae766ad-86df-4df0-b3cf-835a3ae4a55b","Type":"ContainerStarted","Data":"5ae5af450e3c76d4f9700e06883ff15f2d11e6cef05ac1c46ade273359fece8c"}
Jul 10 20:56:47.770217 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:47.770196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" event={"ID":"cae766ad-86df-4df0-b3cf-835a3ae4a55b","Type":"ContainerStarted","Data":"977b4eb19d604fdf5d8316e539372504a201d3afbe20d06b381b72b037588c06"}
Jul 10 20:56:47.770672 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:47.770229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:47.789510 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:47.789447    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" podStartSLOduration=2.36249116 podStartE2EDuration="2.789431438s" totalImagesPullingTime="426.940278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:56:47.788799986 +0000 UTC m=+2131.689105207" watchObservedRunningTime="2026-07-10 20:56:47.789431438 +0000 UTC m=+2131.689736656"
Jul 10 20:56:53.781074 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:53.781039    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:56:58.824247 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:58.824212    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:56:58.824659 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:58.824455    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" podUID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" containerName="maas-api" containerID="cri-o://64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90" gracePeriod=30
Jul 10 20:56:59.068735 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.068711    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:59.072911 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.072896    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") pod \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") "
Jul 10 20:56:59.072977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.072929    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca\") pod \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") "
Jul 10 20:56:59.072977 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.072959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h22b4\" (UniqueName: \"kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4\") pod \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\" (UID: \"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6\") "
Jul 10 20:56:59.073239 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.073216    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" (UID: "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:56:59.074821 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.074772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" (UID: "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:56:59.074897 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.074878    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4" (OuterVolumeSpecName: "kube-api-access-h22b4") pod "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" (UID: "c3f4ae34-bfc9-4546-850a-0df1ed45ddd6"). InnerVolumeSpecName "kube-api-access-h22b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:56:59.174567 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.174543    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:59.174567 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.174564    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:59.174704 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.174573    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22b4\" (UniqueName: \"kubernetes.io/projected/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6-kube-api-access-h22b4\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:56:59.830078 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.830044    2564 generic.go:356] "Generic (PLEG): container finished" podID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" containerID="64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90" exitCode=0
Jul 10 20:56:59.830573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.830107    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"
Jul 10 20:56:59.830573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.830127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" event={"ID":"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6","Type":"ContainerDied","Data":"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"}
Jul 10 20:56:59.830573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.830175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv" event={"ID":"c3f4ae34-bfc9-4546-850a-0df1ed45ddd6","Type":"ContainerDied","Data":"31f130ade1da3d830a3cf4697ecc82bddb1ddc7eaca71cdc836569cb32359ac3"}
Jul 10 20:56:59.830573 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.830199    2564 scope.go:117] "RemoveContainer" containerID="64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"
Jul 10 20:56:59.839554 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.839540    2564 scope.go:117] "RemoveContainer" containerID="64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"
Jul 10 20:56:59.839804 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:56:59.839788    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90\": container with ID starting with 64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90 not found: ID does not exist" containerID="64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"
Jul 10 20:56:59.839860 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.839810    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90"} err="failed to get container status \"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90\": rpc error: code = NotFound desc = could not find container \"64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90\": container with ID starting with 64fb402261365bccc4de53927ebc1de327250f14359955095875e0cd11a35c90 not found: ID does not exist"
Jul 10 20:56:59.852551 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.852531    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:56:59.856803 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:56:59.856784    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec0f24-5c48d5fcf8-cwpbv"]
Jul 10 20:57:00.784470 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:00.784434    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" path="/var/lib/kubelet/pods/c3f4ae34-bfc9-4546-850a-0df1ed45ddd6/volumes"
Jul 10 20:57:02.130110 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:02.130068    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:57:05.723142 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:05.723108    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:57:05.723661 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:05.723352    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" podUID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" containerName="maas-api" containerID="cri-o://5ae5af450e3c76d4f9700e06883ff15f2d11e6cef05ac1c46ade273359fece8c" gracePeriod=30
Jul 10 20:57:05.859137 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:05.859102    2564 generic.go:356] "Generic (PLEG): container finished" podID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" containerID="5ae5af450e3c76d4f9700e06883ff15f2d11e6cef05ac1c46ade273359fece8c" exitCode=0
Jul 10 20:57:05.859274 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:05.859149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" event={"ID":"cae766ad-86df-4df0-b3cf-835a3ae4a55b","Type":"ContainerDied","Data":"5ae5af450e3c76d4f9700e06883ff15f2d11e6cef05ac1c46ade273359fece8c"}
Jul 10 20:57:05.987045 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:05.986990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:57:06.026800 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.026769    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jttjg\" (UniqueName: \"kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg\") pod \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") "
Jul 10 20:57:06.026967 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.026812    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca\") pod \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") "
Jul 10 20:57:06.026967 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.026869    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") pod \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\" (UID: \"cae766ad-86df-4df0-b3cf-835a3ae4a55b\") "
Jul 10 20:57:06.027192 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.027168    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cae766ad-86df-4df0-b3cf-835a3ae4a55b" (UID: "cae766ad-86df-4df0-b3cf-835a3ae4a55b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 20:57:06.028851 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.028818    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg" (OuterVolumeSpecName: "kube-api-access-jttjg") pod "cae766ad-86df-4df0-b3cf-835a3ae4a55b" (UID: "cae766ad-86df-4df0-b3cf-835a3ae4a55b"). InnerVolumeSpecName "kube-api-access-jttjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 20:57:06.028951 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.028897    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cae766ad-86df-4df0-b3cf-835a3ae4a55b" (UID: "cae766ad-86df-4df0-b3cf-835a3ae4a55b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 20:57:06.128060 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.128012    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttjg\" (UniqueName: \"kubernetes.io/projected/cae766ad-86df-4df0-b3cf-835a3ae4a55b-kube-api-access-jttjg\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:57:06.128060 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.128052    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae766ad-86df-4df0-b3cf-835a3ae4a55b-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:57:06.128060 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.128067    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae766ad-86df-4df0-b3cf-835a3ae4a55b-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 20:57:06.864482 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.864439    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4" event={"ID":"cae766ad-86df-4df0-b3cf-835a3ae4a55b","Type":"ContainerDied","Data":"977b4eb19d604fdf5d8316e539372504a201d3afbe20d06b381b72b037588c06"}
Jul 10 20:57:06.864482 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.864467    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"
Jul 10 20:57:06.865016 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.864523    2564 scope.go:117] "RemoveContainer" containerID="5ae5af450e3c76d4f9700e06883ff15f2d11e6cef05ac1c46ade273359fece8c"
Jul 10 20:57:06.883590 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.883570    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:57:06.892781 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:06.892755    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-00c974-66658bb86b-wgns4"]
Jul 10 20:57:08.785041 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:08.784968    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" path="/var/lib/kubelet/pods/cae766ad-86df-4df0-b3cf-835a3ae4a55b/volumes"
Jul 10 20:57:14.939457 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939423    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939779    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" containerName="maas-api"
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" containerName="maas-api"
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939805    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" containerName="maas-api"
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939810    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" containerName="maas-api"
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939864    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae766ad-86df-4df0-b3cf-835a3ae4a55b" containerName="maas-api"
Jul 10 20:57:14.941901 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.939876    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3f4ae34-bfc9-4546-850a-0df1ed45ddd6" containerName="maas-api"
Jul 10 20:57:14.942738 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.942723    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:14.945986 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.945961    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-e560d0-serving-cert\""
Jul 10 20:57:14.951201 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:14.951176    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 20:57:15.001861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.001825    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdhf\" (UniqueName: \"kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.001861 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.001865    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.002079 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.001891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.102675 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.102636    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdhf\" (UniqueName: \"kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.102968 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.102943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.103128 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.103101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.103658 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.103633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.105584 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.105566    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.111265 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.111244    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdhf\" (UniqueName: \"kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf\") pod \"maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.254102 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.254015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.397076 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.397051    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 20:57:15.399016 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:57:15.398989    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7038d7b9_eecf_4ac3_a93c_ad078207bf07.slice/crio-4a4cd02a99bc0b953fbca25f3f3beaeb7ac6aba411c074e700a6b575081bdadd WatchSource:0}: Error finding container 4a4cd02a99bc0b953fbca25f3f3beaeb7ac6aba411c074e700a6b575081bdadd: Status 404 returned error can't find the container with id 4a4cd02a99bc0b953fbca25f3f3beaeb7ac6aba411c074e700a6b575081bdadd
Jul 10 20:57:15.439528 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.439476    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:57:15.909900 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.909858    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" event={"ID":"7038d7b9-eecf-4ac3-a93c-ad078207bf07","Type":"ContainerStarted","Data":"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"}
Jul 10 20:57:15.910108 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.909935    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:15.910108 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.909964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" event={"ID":"7038d7b9-eecf-4ac3-a93c-ad078207bf07","Type":"ContainerStarted","Data":"4a4cd02a99bc0b953fbca25f3f3beaeb7ac6aba411c074e700a6b575081bdadd"}
Jul 10 20:57:15.931172 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:15.931120    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" podStartSLOduration=1.590086128 podStartE2EDuration="1.93110285s" totalImagesPullingTime="341.016722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:57:15.931046097 +0000 UTC m=+2159.831351349" watchObservedRunningTime="2026-07-10 20:57:15.93110285 +0000 UTC m=+2159.831408100"
Jul 10 20:57:21.920319 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:21.920290    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 20:57:23.116883 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.116844    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 20:57:23.122576 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.122542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.125748 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.125725    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-28db1d-serving-cert\""
Jul 10 20:57:23.131885 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.131859    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 20:57:23.164381 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.164342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q747\" (UniqueName: \"kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.164539 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.164391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.164539 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.164503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.265471 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.265442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.265597 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.265479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q747\" (UniqueName: \"kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.265597 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.265515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.265597 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:57:23.265580    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-serving-cert: secret "maas-api-e2e-shared-b-28db1d-serving-cert" not found
Jul 10 20:57:23.265701 ip-10-0-141-240 kubenswrapper[2564]: E0710 20:57:23.265648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls podName:762bcf66-74ff-4637-9b78-a84c4184f446 nodeName:}" failed. No retries permitted until 2026-07-10 20:57:23.765631766 +0000 UTC m=+2167.665936963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls") pod "maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" (UID: "762bcf66-74ff-4637-9b78-a84c4184f446") : secret "maas-api-e2e-shared-b-28db1d-serving-cert" not found
Jul 10 20:57:23.266112 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.266093    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.274609 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.274589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q747\" (UniqueName: \"kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.768694 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.768657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:23.770994 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:23.770968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") pod \"maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:24.035894 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.035811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:24.162805 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.162778    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 20:57:24.164519 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:57:24.164475    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod762bcf66_74ff_4637_9b78_a84c4184f446.slice/crio-45cf6efffc56b571c48e6116851a7ba60b7b6022ca2d25cc2b5b3db9b3468aac WatchSource:0}: Error finding container 45cf6efffc56b571c48e6116851a7ba60b7b6022ca2d25cc2b5b3db9b3468aac: Status 404 returned error can't find the container with id 45cf6efffc56b571c48e6116851a7ba60b7b6022ca2d25cc2b5b3db9b3468aac
Jul 10 20:57:24.953853 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.953772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" event={"ID":"762bcf66-74ff-4637-9b78-a84c4184f446","Type":"ContainerStarted","Data":"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"}
Jul 10 20:57:24.953853 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.953822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" event={"ID":"762bcf66-74ff-4637-9b78-a84c4184f446","Type":"ContainerStarted","Data":"45cf6efffc56b571c48e6116851a7ba60b7b6022ca2d25cc2b5b3db9b3468aac"}
Jul 10 20:57:24.953853 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.953851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:24.972276 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:24.972221    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" podStartSLOduration=1.533679214 podStartE2EDuration="1.972205111s" totalImagesPullingTime="438.525897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:57:24.970183034 +0000 UTC m=+2168.870488255" watchObservedRunningTime="2026-07-10 20:57:24.972205111 +0000 UTC m=+2168.872510330"
Jul 10 20:57:28.620276 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:28.620241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:57:30.963460 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:30.963431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 20:57:38.623029 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.622994    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 20:57:38.626517 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.626480    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.629505 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.629473    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"kube-root-ca.crt\""
Jul 10 20:57:38.630773 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.630755    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"openshift-service-ca.crt\""
Jul 10 20:57:38.630862 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.630764    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"default-dockercfg-z4mkw\""
Jul 10 20:57:38.630941 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.630920    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"auth-test-model-e560d0-kserve-self-signed-certs\""
Jul 10 20:57:38.636100 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.636077    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 20:57:38.781765 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781738    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.781928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.781928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781807    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.781928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.781928 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.782094 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.781938    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcphj\" (UniqueName: \"kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882512 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcphj\" (UniqueName: \"kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882512 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882621    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882714 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.882989 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.882968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.883069 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.883018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.883069 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.883055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.884652 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.884629    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.884964 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.884947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.890174 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.890153    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcphj\" (UniqueName: \"kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj\") pod \"auth-test-model-e560d0-kserve-66d7cc56f5-trzjx\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") " pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:38.937168 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:38.937140    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:39.066807 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:39.066782    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 20:57:39.068422 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:57:39.068392    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fe23978_e8f1_4719_8273_cdb075b95ce2.slice/crio-1ad3c76174121a2d20634fedd62cb0914736824fac35645cad180edc29dd11a1 WatchSource:0}: Error finding container 1ad3c76174121a2d20634fedd62cb0914736824fac35645cad180edc29dd11a1: Status 404 returned error can't find the container with id 1ad3c76174121a2d20634fedd62cb0914736824fac35645cad180edc29dd11a1
Jul 10 20:57:40.020443 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:40.020407    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" event={"ID":"4fe23978-e8f1-4719-8273-cdb075b95ce2","Type":"ContainerStarted","Data":"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"}
Jul 10 20:57:40.020807 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:40.020458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" event={"ID":"4fe23978-e8f1-4719-8273-cdb075b95ce2","Type":"ContainerStarted","Data":"1ad3c76174121a2d20634fedd62cb0914736824fac35645cad180edc29dd11a1"}
Jul 10 20:57:40.038264 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:40.038221    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" podStartSLOduration=2.038207526 podStartE2EDuration="2.038207526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:57:40.037007174 +0000 UTC m=+2183.937312394" watchObservedRunningTime="2026-07-10 20:57:40.038207526 +0000 UTC m=+2183.938512744"
Jul 10 20:57:42.528304 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:42.528270    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:57:48.938117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:48.938075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:48.938117 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:48.938130    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:48.950555 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:48.950529    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:57:49.074176 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:57:49.074149    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 20:58:38.620187 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.620106    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 20:58:38.623414 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.623393    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.626240 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.626219    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"kube-root-ca.crt\""
Jul 10 20:58:38.626352 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.626239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"openshift-service-ca.crt\""
Jul 10 20:58:38.627674 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.627654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"default-dockercfg-rhdwx\""
Jul 10 20:58:38.627795 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.627677    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"auth-test-model-28db1d-kserve-self-signed-certs\""
Jul 10 20:58:38.635832 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.635812    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 20:58:38.729988 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.729961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.730118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.729990    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.730118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.730062    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfzst\" (UniqueName: \"kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.730118 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.730086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.730345 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.730190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.730345 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.730246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.830830 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.830802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.830830 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.830831    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831000 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.830865    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfzst\" (UniqueName: \"kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831000 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.830900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831140 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.831113    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831302 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.831223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831419 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.831339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831569 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.831461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.831673 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.831615    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.833165 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.833146    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.833421 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.833402    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.839619 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.839595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfzst\" (UniqueName: \"kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst\") pod \"auth-test-model-28db1d-kserve-86bcdb576c-gx9xq\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") " pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:38.932537 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:38.932444    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:39.062810 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:39.062783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 20:58:39.064457 ip-10-0-141-240 kubenswrapper[2564]: W0710 20:58:39.064418    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a5201ec_ec4c_46e2_b7e1_6f1fd244ad56.slice/crio-0e94bc2962d0113815d156ea910a3dcec25d145dcc4ac99e7a6045ecf4958a67 WatchSource:0}: Error finding container 0e94bc2962d0113815d156ea910a3dcec25d145dcc4ac99e7a6045ecf4958a67: Status 404 returned error can't find the container with id 0e94bc2962d0113815d156ea910a3dcec25d145dcc4ac99e7a6045ecf4958a67
Jul 10 20:58:39.296553 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:39.296450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" event={"ID":"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56","Type":"ContainerStarted","Data":"d70702cf87d42393f998a671c6ce18a99e108bf1cba57c7632df1e36c9964f27"}
Jul 10 20:58:39.296553 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:39.296514    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" event={"ID":"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56","Type":"ContainerStarted","Data":"0e94bc2962d0113815d156ea910a3dcec25d145dcc4ac99e7a6045ecf4958a67"}
Jul 10 20:58:39.315729 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:39.315682    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" podStartSLOduration=1.315668101 podStartE2EDuration="1.315668101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 20:58:39.314933092 +0000 UTC m=+2243.215238302" watchObservedRunningTime="2026-07-10 20:58:39.315668101 +0000 UTC m=+2243.215973320"
Jul 10 20:58:42.317997 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:42.317964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:58:48.932734 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:48.932705    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:48.932734 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:48.932744    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:48.945125 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:48.945102    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:49.359398 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:49.359365    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 20:58:57.028965 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:58:57.028930    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:59:15.125983 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:59:15.125950    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:59:31.233107 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:59:31.233070    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 20:59:48.034254 ip-10-0-141-240 kubenswrapper[2564]: I0710 20:59:48.034220    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:00:04.426197 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:04.426164    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:00:22.022406 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:22.022322    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:00:33.821171 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:33.821130    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 21:00:34.704619 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:34.704586    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 21:00:39.220519 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:39.220474    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:00:48.842506 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:48.842441    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerName="main" containerID="cri-o://7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e" gracePeriod=45
Jul 10 21:00:49.085535 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.085512    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 21:00:49.191591 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191591 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191796 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191602    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191796 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191796 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191677    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcphj\" (UniqueName: \"kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191796 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191709    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm\") pod \"4fe23978-e8f1-4719-8273-cdb075b95ce2\" (UID: \"4fe23978-e8f1-4719-8273-cdb075b95ce2\") "
Jul 10 21:00:49.191995 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191835    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home" (OuterVolumeSpecName: "home") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:49.191995 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191851    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:49.191995 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache" (OuterVolumeSpecName: "model-cache") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:49.191995 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191939    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.191995 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191981    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.192158 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.191997    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.193854 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.193823    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj" (OuterVolumeSpecName: "kube-api-access-kcphj") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "kube-api-access-kcphj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:00:49.193954 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.193866    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:00:49.193954 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.193863    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm" (OuterVolumeSpecName: "dshm") pod "4fe23978-e8f1-4719-8273-cdb075b95ce2" (UID: "4fe23978-e8f1-4719-8273-cdb075b95ce2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:49.292682 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.292644    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcphj\" (UniqueName: \"kubernetes.io/projected/4fe23978-e8f1-4719-8273-cdb075b95ce2-kube-api-access-kcphj\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.292682 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.292680    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe23978-e8f1-4719-8273-cdb075b95ce2-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.292682 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.292692    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe23978-e8f1-4719-8273-cdb075b95ce2-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:49.726929 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.726866    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" podUID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" containerName="main" containerID="cri-o://d70702cf87d42393f998a671c6ce18a99e108bf1cba57c7632df1e36c9964f27" gracePeriod=45
Jul 10 21:00:49.911380 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.911347    2564 generic.go:356] "Generic (PLEG): container finished" podID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerID="7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e" exitCode=0
Jul 10 21:00:49.911784 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.911392    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"
Jul 10 21:00:49.911784 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.911428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" event={"ID":"4fe23978-e8f1-4719-8273-cdb075b95ce2","Type":"ContainerDied","Data":"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"}
Jul 10 21:00:49.911784 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.911476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" event={"ID":"4fe23978-e8f1-4719-8273-cdb075b95ce2","Type":"ContainerDied","Data":"1ad3c76174121a2d20634fedd62cb0914736824fac35645cad180edc29dd11a1"}
Jul 10 21:00:49.911784 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.911510    2564 scope.go:117] "RemoveContainer" containerID="7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"
Jul 10 21:00:49.913955 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.913932    2564 generic.go:356] "Generic (PLEG): container finished" podID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" containerID="d70702cf87d42393f998a671c6ce18a99e108bf1cba57c7632df1e36c9964f27" exitCode=0
Jul 10 21:00:49.914073 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.913983    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" event={"ID":"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56","Type":"ContainerDied","Data":"d70702cf87d42393f998a671c6ce18a99e108bf1cba57c7632df1e36c9964f27"}
Jul 10 21:00:49.921027 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.921003    2564 scope.go:117] "RemoveContainer" containerID="7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"
Jul 10 21:00:49.921309 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:00:49.921290    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e\": container with ID starting with 7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e not found: ID does not exist" containerID="7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"
Jul 10 21:00:49.921384 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.921315    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e"} err="failed to get container status \"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e\": rpc error: code = NotFound desc = could not find container \"7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e\": container with ID starting with 7db292e4ef87326193e8b035de0602a057989eee29c5ee66484669ff4f95bb1e not found: ID does not exist"
Jul 10 21:00:49.935524 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.935502    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 21:00:49.940204 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.940183    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx"]
Jul 10 21:00:49.975275 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:49.975259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 21:00:50.098072 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098048    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098217 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098217 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098123    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098217 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098217 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098212    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098245    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfzst\" (UniqueName: \"kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst\") pod \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\" (UID: \"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56\") "
Jul 10 21:00:50.098416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098375    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:50.098545 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098437    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:50.098601 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098553    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.098601 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098555    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home" (OuterVolumeSpecName: "home") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:50.098601 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.098572    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.100282 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.100253    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm" (OuterVolumeSpecName: "dshm") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:00:50.100374 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.100338    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst" (OuterVolumeSpecName: "kube-api-access-vfzst") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "kube-api-access-vfzst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:00:50.100374 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.100361    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" (UID: "0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:00:50.199435 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.199412    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.199435 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.199433    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfzst\" (UniqueName: \"kubernetes.io/projected/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-kube-api-access-vfzst\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.199589 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.199443    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.199589 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.199451    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:00:50.784476 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.784431    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" path="/var/lib/kubelet/pods/4fe23978-e8f1-4719-8273-cdb075b95ce2/volumes"
Jul 10 21:00:50.923778 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.923748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq" event={"ID":"0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56","Type":"ContainerDied","Data":"0e94bc2962d0113815d156ea910a3dcec25d145dcc4ac99e7a6045ecf4958a67"}
Jul 10 21:00:50.923778 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.923772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"
Jul 10 21:00:50.923778 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.923792    2564 scope.go:117] "RemoveContainer" containerID="d70702cf87d42393f998a671c6ce18a99e108bf1cba57c7632df1e36c9964f27"
Jul 10 21:00:50.942996 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.942974    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 21:00:50.944636 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:50.944618    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/auth-test-model-28db1d-kserve-86bcdb576c-gx9xq"]
Jul 10 21:00:52.785061 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:52.785029    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" path="/var/lib/kubelet/pods/0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56/volumes"
Jul 10 21:00:54.063676 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:54.063636    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-e560d0/auth-test-model-e560d0-kserve-66d7cc56f5-trzjx" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 10 21:00:55.122876 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.122835    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:00:55.123249 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123183    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerName="main"
Jul 10 21:00:55.123249 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123193    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerName="main"
Jul 10 21:00:55.123249 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123207    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" containerName="main"
Jul 10 21:00:55.123249 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123212    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" containerName="main"
Jul 10 21:00:55.123399 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123269    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a5201ec-ec4c-46e2-b7e1-6f1fd244ad56" containerName="main"
Jul 10 21:00:55.123399 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.123281    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe23978-e8f1-4719-8273-cdb075b95ce2" containerName="main"
Jul 10 21:00:55.126370 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.126348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.130641 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.130623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"kube-root-ca.crt\""
Jul 10 21:00:55.130641 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.130634    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"sub-test-model-e560d0-kserve-self-signed-certs\""
Jul 10 21:00:55.130788 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.130624    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"openshift-service-ca.crt\""
Jul 10 21:00:55.130788 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.130657    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"default-dockercfg-z4mkw\""
Jul 10 21:00:55.137228 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.137207    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:00:55.241354 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.241535 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241364    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.241535 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.241535 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.241674 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.241674 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.241568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scgp\" (UniqueName: \"kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342508 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342508 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342758 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342535    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scgp\" (UniqueName: \"kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342758 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342758 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.342758 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.342737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.343077 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.343055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.343159 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.343080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.343159 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.343101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.344703 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.344678    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.344944 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.344929    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.356934 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.356904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scgp\" (UniqueName: \"kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp\") pod \"sub-test-model-e560d0-kserve-557cf7867f-px2hc\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") " pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.435517 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.435426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:00:55.565248 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.565221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:00:55.568147 ip-10-0-141-240 kubenswrapper[2564]: W0710 21:00:55.568114    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5ace3f5_f4a5_4343_b944_060419f6e9e5.slice/crio-6712bcf61d8b247907b8ffe19361c02762f398fccc667d02b48562de794ca5bd WatchSource:0}: Error finding container 6712bcf61d8b247907b8ffe19361c02762f398fccc667d02b48562de794ca5bd: Status 404 returned error can't find the container with id 6712bcf61d8b247907b8ffe19361c02762f398fccc667d02b48562de794ca5bd
Jul 10 21:00:55.950977 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.950936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" event={"ID":"c5ace3f5-f4a5-4343-b944-060419f6e9e5","Type":"ContainerStarted","Data":"b9887ae3d6e33bf0cfbc03a2119a4f37b75257c5465ef7bfa9ed9b730ad9fbae"}
Jul 10 21:00:55.951141 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.950986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" event={"ID":"c5ace3f5-f4a5-4343-b944-060419f6e9e5","Type":"ContainerStarted","Data":"6712bcf61d8b247907b8ffe19361c02762f398fccc667d02b48562de794ca5bd"}
Jul 10 21:00:55.967587 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:55.967537    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" podStartSLOduration=0.967521648 podStartE2EDuration="967.521648ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:00:55.966974124 +0000 UTC m=+2379.867279354" watchObservedRunningTime="2026-07-10 21:00:55.967521648 +0000 UTC m=+2379.867826868"
Jul 10 21:00:59.045113 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:00:59.045080    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:01:05.435881 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:05.435851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:01:05.435881 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:05.435890    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:01:05.448424 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:05.448393    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:01:06.016212 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:06.016178    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:01:14.926908 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.926876    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:01:14.930568 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.930545    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:14.934876 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.934849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"kube-root-ca.crt\""
Jul 10 21:01:14.934876 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.934864    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"openshift-service-ca.crt\""
Jul 10 21:01:14.935077 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.934891    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"default-dockercfg-rhdwx\""
Jul 10 21:01:14.935077 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.934892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"sub-test-model-28db1d-kserve-self-signed-certs\""
Jul 10 21:01:14.938626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:14.938465    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:01:15.114886 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.114850    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.115102 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.114902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.115102 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.114932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.115102 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.114979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4fx\" (UniqueName: \"kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.115102 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.115010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.115102 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.115037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.215957 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.215859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.215957 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.215914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.215957 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.215939    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216256 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.215990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4fx\" (UniqueName: \"kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216256 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.216019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216256 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.216048    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.216379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.216388    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.216416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.216406    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.218318 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.218298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.218451 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.218435    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.223969 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.223947    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4fx\" (UniqueName: \"kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx\") pod \"sub-test-model-28db1d-kserve-64cc6bd897-wfdcv\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") " pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.241865 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.241828    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:15.380726 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:15.380698    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:01:15.382327 ip-10-0-141-240 kubenswrapper[2564]: W0710 21:01:15.382300    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cdd78e0_8576_4fb7_807e_94d269063979.slice/crio-14fe0c84dce349ed0f28a2f2bbfd87779e7b3201758d1157e61130e11675e4b2 WatchSource:0}: Error finding container 14fe0c84dce349ed0f28a2f2bbfd87779e7b3201758d1157e61130e11675e4b2: Status 404 returned error can't find the container with id 14fe0c84dce349ed0f28a2f2bbfd87779e7b3201758d1157e61130e11675e4b2
Jul 10 21:01:16.052583 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:16.052549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" event={"ID":"1cdd78e0-8576-4fb7-807e-94d269063979","Type":"ContainerStarted","Data":"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"}
Jul 10 21:01:16.052981 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:16.052596    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" event={"ID":"1cdd78e0-8576-4fb7-807e-94d269063979","Type":"ContainerStarted","Data":"14fe0c84dce349ed0f28a2f2bbfd87779e7b3201758d1157e61130e11675e4b2"}
Jul 10 21:01:16.070537 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:16.070468    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" podStartSLOduration=2.070452179 podStartE2EDuration="2.070452179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:01:16.069765519 +0000 UTC m=+2399.970070739" watchObservedRunningTime="2026-07-10 21:01:16.070452179 +0000 UTC m=+2399.970757398"
Jul 10 21:01:16.794209 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:16.794172    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 21:01:16.807181 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:16.807149    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fqfdp_22863a35-4dda-44cb-ae0a-b922feaed80b/ovn-acl-logging/0.log"
Jul 10 21:01:19.725057 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:19.725024    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:01:25.242603 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:25.242561    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:25.243106 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:25.242616    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:25.255223 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:25.255194    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:26.112419 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:26.112390    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:01:42.629733 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:42.629655    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:01:58.521354 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:01:58.521316    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:02:04.215084 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:04.215051    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:02:05.016962 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:05.016928    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:02:10.522959 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.522920    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:02:10.526606 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.526579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.529576 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.529551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-e560d0\"/\"rate-test-model-e560d0-kserve-self-signed-certs\""
Jul 10 21:02:10.534733 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.534705    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:02:10.649272 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.649272 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649275    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.649578 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649335    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.649578 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.649578 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649481    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbncm\" (UniqueName: \"kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.649578 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.649549    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750556 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbncm\" (UniqueName: \"kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750556 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750599    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.750847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.750668    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.751164 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.751141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.751235 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.751176    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.751235 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.751201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.752851 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.752831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.753113 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.753094    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.759637 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.759613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbncm\" (UniqueName: \"kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm\") pod \"rate-test-model-e560d0-kserve-6d9457867f-sgn8p\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") " pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.837635 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.837601    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:10.966847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:10.966824    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:02:10.969669 ip-10-0-141-240 kubenswrapper[2564]: W0710 21:02:10.969629    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3c4b5eb_8927_490d_b0cb_857fdccd5867.slice/crio-492bdebabe30cf0ffa232e4a70123a201feb39de73a80149d6939ffb3a0b7043 WatchSource:0}: Error finding container 492bdebabe30cf0ffa232e4a70123a201feb39de73a80149d6939ffb3a0b7043: Status 404 returned error can't find the container with id 492bdebabe30cf0ffa232e4a70123a201feb39de73a80149d6939ffb3a0b7043
Jul 10 21:02:11.312960 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:11.312921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" event={"ID":"d3c4b5eb-8927-490d-b0cb-857fdccd5867","Type":"ContainerStarted","Data":"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"}
Jul 10 21:02:11.313131 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:11.312970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" event={"ID":"d3c4b5eb-8927-490d-b0cb-857fdccd5867","Type":"ContainerStarted","Data":"492bdebabe30cf0ffa232e4a70123a201feb39de73a80149d6939ffb3a0b7043"}
Jul 10 21:02:11.330538 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:11.330472    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" podStartSLOduration=1.330455207 podStartE2EDuration="1.330455207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:02:11.329298823 +0000 UTC m=+2455.229604057" watchObservedRunningTime="2026-07-10 21:02:11.330455207 +0000 UTC m=+2455.230760427"
Jul 10 21:02:17.535849 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:17.535811    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:02:19.234745 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.234704    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" podUID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" containerName="main" containerID="cri-o://b9887ae3d6e33bf0cfbc03a2119a4f37b75257c5465ef7bfa9ed9b730ad9fbae" gracePeriod=45
Jul 10 21:02:19.351934 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.351899    2564 generic.go:356] "Generic (PLEG): container finished" podID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" containerID="b9887ae3d6e33bf0cfbc03a2119a4f37b75257c5465ef7bfa9ed9b730ad9fbae" exitCode=0
Jul 10 21:02:19.352056 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.351938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" event={"ID":"c5ace3f5-f4a5-4343-b944-060419f6e9e5","Type":"ContainerDied","Data":"b9887ae3d6e33bf0cfbc03a2119a4f37b75257c5465ef7bfa9ed9b730ad9fbae"}
Jul 10 21:02:19.488558 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.488476    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:02:19.625255 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625222    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625262    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625334    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625416 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7scgp\" (UniqueName: \"kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625645 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625452    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs\") pod \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\" (UID: \"c5ace3f5-f4a5-4343-b944-060419f6e9e5\") "
Jul 10 21:02:19.625645 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625519    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:19.625645 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625566    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:19.625764 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625663    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home" (OuterVolumeSpecName: "home") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:19.625812 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625788    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:19.625812 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625805    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:19.625920 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.625813    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:19.627277 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.627257    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm" (OuterVolumeSpecName: "dshm") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:19.627463 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.627441    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp" (OuterVolumeSpecName: "kube-api-access-7scgp") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "kube-api-access-7scgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:02:19.627563 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.627542    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5ace3f5-f4a5-4343-b944-060419f6e9e5" (UID: "c5ace3f5-f4a5-4343-b944-060419f6e9e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:02:19.726763 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.726742    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ace3f5-f4a5-4343-b944-060419f6e9e5-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:19.726763 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.726762    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7scgp\" (UniqueName: \"kubernetes.io/projected/c5ace3f5-f4a5-4343-b944-060419f6e9e5-kube-api-access-7scgp\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:19.726915 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:19.726773    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ace3f5-f4a5-4343-b944-060419f6e9e5-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.037229 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.037174    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" podUID="1cdd78e0-8576-4fb7-807e-94d269063979" containerName="main" containerID="cri-o://446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18" gracePeriod=45
Jul 10 21:02:20.300323 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.300268    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:02:20.357992 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.357967    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"
Jul 10 21:02:20.358124 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.357967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc" event={"ID":"c5ace3f5-f4a5-4343-b944-060419f6e9e5","Type":"ContainerDied","Data":"6712bcf61d8b247907b8ffe19361c02762f398fccc667d02b48562de794ca5bd"}
Jul 10 21:02:20.358124 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.358096    2564 scope.go:117] "RemoveContainer" containerID="b9887ae3d6e33bf0cfbc03a2119a4f37b75257c5465ef7bfa9ed9b730ad9fbae"
Jul 10 21:02:20.359542 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.359513    2564 generic.go:356] "Generic (PLEG): container finished" podID="1cdd78e0-8576-4fb7-807e-94d269063979" containerID="446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18" exitCode=0
Jul 10 21:02:20.359677 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.359540    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" event={"ID":"1cdd78e0-8576-4fb7-807e-94d269063979","Type":"ContainerDied","Data":"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"}
Jul 10 21:02:20.359677 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.359559    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"
Jul 10 21:02:20.359677 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.359590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv" event={"ID":"1cdd78e0-8576-4fb7-807e-94d269063979","Type":"ContainerDied","Data":"14fe0c84dce349ed0f28a2f2bbfd87779e7b3201758d1157e61130e11675e4b2"}
Jul 10 21:02:20.368612 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.368593    2564 scope.go:117] "RemoveContainer" containerID="446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"
Jul 10 21:02:20.376582 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.376564    2564 scope.go:117] "RemoveContainer" containerID="446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"
Jul 10 21:02:20.376821 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:02:20.376803    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18\": container with ID starting with 446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18 not found: ID does not exist" containerID="446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"
Jul 10 21:02:20.376879 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.376830    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18"} err="failed to get container status \"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18\": rpc error: code = NotFound desc = could not find container \"446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18\": container with ID starting with 446d5b41db7b05da5ebb6e9920d0f9fe454b90582ab5db3777532384ce6b3f18 not found: ID does not exist"
Jul 10 21:02:20.383145 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.383124    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:02:20.386945 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.386926    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/sub-test-model-e560d0-kserve-557cf7867f-px2hc"]
Jul 10 21:02:20.432586 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432568    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432686 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432610    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf4fx\" (UniqueName: \"kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432686 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432649    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432686 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432672    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432848 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432810    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432907 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432885    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs\") pod \"1cdd78e0-8576-4fb7-807e-94d269063979\" (UID: \"1cdd78e0-8576-4fb7-807e-94d269063979\") "
Jul 10 21:02:20.432960 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432927    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:20.432960 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.432945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:20.433052 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.433033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home" (OuterVolumeSpecName: "home") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:20.433206 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.433186    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.433307 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.433212    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.433307 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.433225    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.434515 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.434480    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm" (OuterVolumeSpecName: "dshm") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:02:20.435191 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.435172    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx" (OuterVolumeSpecName: "kube-api-access-rf4fx") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "kube-api-access-rf4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:02:20.435251 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.435171    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cdd78e0-8576-4fb7-807e-94d269063979" (UID: "1cdd78e0-8576-4fb7-807e-94d269063979"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:02:20.534134 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.534109    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd78e0-8576-4fb7-807e-94d269063979-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.534134 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.534130    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd78e0-8576-4fb7-807e-94d269063979-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.534268 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.534140    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf4fx\" (UniqueName: \"kubernetes.io/projected/1cdd78e0-8576-4fb7-807e-94d269063979-kube-api-access-rf4fx\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:02:20.684329 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.684301    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:02:20.688377 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.688354    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/sub-test-model-28db1d-kserve-64cc6bd897-wfdcv"]
Jul 10 21:02:20.784851 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.784824    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdd78e0-8576-4fb7-807e-94d269063979" path="/var/lib/kubelet/pods/1cdd78e0-8576-4fb7-807e-94d269063979/volumes"
Jul 10 21:02:20.785214 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.785202    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" path="/var/lib/kubelet/pods/c5ace3f5-f4a5-4343-b944-060419f6e9e5/volumes"
Jul 10 21:02:20.838151 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.838129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:20.838249 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.838173    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:20.850289 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:20.850272    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:21.377331 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:21.377304    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:02:44.312285 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312241    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:02:44.312865 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312846    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdd78e0-8576-4fb7-807e-94d269063979" containerName="main"
Jul 10 21:02:44.312946 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312867    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdd78e0-8576-4fb7-807e-94d269063979" containerName="main"
Jul 10 21:02:44.312946 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312890    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" containerName="main"
Jul 10 21:02:44.312946 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312898    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" containerName="main"
Jul 10 21:02:44.313103 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.312981    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdd78e0-8576-4fb7-807e-94d269063979" containerName="main"
Jul 10 21:02:44.313103 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.313000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ace3f5-f4a5-4343-b944-060419f6e9e5" containerName="main"
Jul 10 21:02:44.316381 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.316355    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.320819 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.320792    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"rate-test-model-28db1d-kserve-self-signed-certs\""
Jul 10 21:02:44.320943 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.320825    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"kube-root-ca.crt\""
Jul 10 21:02:44.320943 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.320848    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"openshift-service-ca.crt\""
Jul 10 21:02:44.320943 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.320876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-28db1d\"/\"default-dockercfg-rhdwx\""
Jul 10 21:02:44.326858 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.326835    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:02:44.411568 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.411761 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411589    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.411761 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.411761 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.411915 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411766    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.411915 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.411814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndrml\" (UniqueName: \"kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.512981 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.512938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.512981 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.512999    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513238 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513025    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513238 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513238 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513062    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513238 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513090    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndrml\" (UniqueName: \"kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513549 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513528    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513642 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513564    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.513642 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.513598    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.515196 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.515180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.515556 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.515540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.520914 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.520894    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndrml\" (UniqueName: \"kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml\") pod \"rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") " pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.626032 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.625952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:44.762250 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:44.762222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:02:44.763792 ip-10-0-141-240 kubenswrapper[2564]: W0710 21:02:44.763761    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f46b02b_65b9_4c64_b93d_520508225087.slice/crio-929ed025771361d8300a431310addf8e372496bc1aaf08c19ec1e5afd785428b WatchSource:0}: Error finding container 929ed025771361d8300a431310addf8e372496bc1aaf08c19ec1e5afd785428b: Status 404 returned error can't find the container with id 929ed025771361d8300a431310addf8e372496bc1aaf08c19ec1e5afd785428b
Jul 10 21:02:45.477903 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:45.477871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" event={"ID":"8f46b02b-65b9-4c64-b93d-520508225087","Type":"ContainerStarted","Data":"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"}
Jul 10 21:02:45.478264 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:45.477910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" event={"ID":"8f46b02b-65b9-4c64-b93d-520508225087","Type":"ContainerStarted","Data":"929ed025771361d8300a431310addf8e372496bc1aaf08c19ec1e5afd785428b"}
Jul 10 21:02:45.494780 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:45.494733    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" podStartSLOduration=1.494719996 podStartE2EDuration="1.494719996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 21:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:02:45.494323724 +0000 UTC m=+2489.394628941" watchObservedRunningTime="2026-07-10 21:02:45.494719996 +0000 UTC m=+2489.395025217"
Jul 10 21:02:49.025047 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:49.025004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:02:54.626095 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:54.626062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:54.626095 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:54.626100    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:54.638504 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:54.638466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:02:55.538343 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:02:55.538307    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:03:03.623764 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:03.623726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:03:19.836328 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:19.836297    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:03:28.008572 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:28.008539    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:03:28.795110 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:28.795077    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:03:34.973507 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:34.973458    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:03:43.702524 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:43.702442    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" podUID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" containerName="main" containerID="cri-o://2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63" gracePeriod=45
Jul 10 21:03:43.731358 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:03:43.731328    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3c4b5eb_8927_490d_b0cb_857fdccd5867.slice/crio-conmon-2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63.scope\": RecentStats: unable to find data in memory cache]"
Jul 10 21:03:43.815032 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:43.814977    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" podUID="8f46b02b-65b9-4c64-b93d-520508225087" containerName="main" containerID="cri-o://298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1" gracePeriod=45
Jul 10 21:03:43.948950 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:43.948923    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:03:44.080243 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.080214    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:03:44.082334 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082316    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082413 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082347    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082413 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082371    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082528 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082436    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082587 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082551    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082587 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082578    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbncm\" (UniqueName: \"kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm\") pod \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\" (UID: \"d3c4b5eb-8927-490d-b0cb-857fdccd5867\") "
Jul 10 21:03:44.082688 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082596    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.082786 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082762    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home" (OuterVolumeSpecName: "home") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.082841 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.082926 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082907    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.082980 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082932    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.082980 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.082945    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.084401 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.084380    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm" (OuterVolumeSpecName: "dshm") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.084467 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.084430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:03:44.084544 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.084513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm" (OuterVolumeSpecName: "kube-api-access-gbncm") pod "d3c4b5eb-8927-490d-b0cb-857fdccd5867" (UID: "d3c4b5eb-8927-490d-b0cb-857fdccd5867"). InnerVolumeSpecName "kube-api-access-gbncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:03:44.183621 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183587    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ndrml\" (UniqueName: \"kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.183621 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.183847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183661    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.183847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183735    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.183847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183775    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.183847 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183799    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache\") pod \"8f46b02b-65b9-4c64-b93d-520508225087\" (UID: \"8f46b02b-65b9-4c64-b93d-520508225087\") "
Jul 10 21:03:44.184045 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.183975    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.184099 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184030    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home" (OuterVolumeSpecName: "home") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.184153 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184095    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3c4b5eb-8927-490d-b0cb-857fdccd5867-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.184153 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184113    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3c4b5eb-8927-490d-b0cb-857fdccd5867-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.184153 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184128    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-tmp-dir\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.184300 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184164    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbncm\" (UniqueName: \"kubernetes.io/projected/d3c4b5eb-8927-490d-b0cb-857fdccd5867-kube-api-access-gbncm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.184300 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.184198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.185757 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.185733    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm" (OuterVolumeSpecName: "dshm") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 10 21:03:44.185757 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.185752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml" (OuterVolumeSpecName: "kube-api-access-ndrml") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "kube-api-access-ndrml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:03:44.185913 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.185843    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f46b02b-65b9-4c64-b93d-520508225087" (UID: "8f46b02b-65b9-4c64-b93d-520508225087"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:03:44.285357 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.285243    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-home\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.285357 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.285273    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f46b02b-65b9-4c64-b93d-520508225087-tls-certs\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.285357 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.285286    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-model-cache\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.285357 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.285298    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndrml\" (UniqueName: \"kubernetes.io/projected/8f46b02b-65b9-4c64-b93d-520508225087-kube-api-access-ndrml\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.285357 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.285310    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f46b02b-65b9-4c64-b93d-520508225087-dshm\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:44.760619 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.760589    2564 generic.go:356] "Generic (PLEG): container finished" podID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" containerID="2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63" exitCode=0
Jul 10 21:03:44.761055 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.760632    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"
Jul 10 21:03:44.761055 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.760685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" event={"ID":"d3c4b5eb-8927-490d-b0cb-857fdccd5867","Type":"ContainerDied","Data":"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"}
Jul 10 21:03:44.761055 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.760736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p" event={"ID":"d3c4b5eb-8927-490d-b0cb-857fdccd5867","Type":"ContainerDied","Data":"492bdebabe30cf0ffa232e4a70123a201feb39de73a80149d6939ffb3a0b7043"}
Jul 10 21:03:44.761055 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.760763    2564 scope.go:117] "RemoveContainer" containerID="2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"
Jul 10 21:03:44.762587 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.762553    2564 generic.go:356] "Generic (PLEG): container finished" podID="8f46b02b-65b9-4c64-b93d-520508225087" containerID="298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1" exitCode=0
Jul 10 21:03:44.762699 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.762607    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"
Jul 10 21:03:44.762699 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.762628    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" event={"ID":"8f46b02b-65b9-4c64-b93d-520508225087","Type":"ContainerDied","Data":"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"}
Jul 10 21:03:44.762699 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.762657    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc" event={"ID":"8f46b02b-65b9-4c64-b93d-520508225087","Type":"ContainerDied","Data":"929ed025771361d8300a431310addf8e372496bc1aaf08c19ec1e5afd785428b"}
Jul 10 21:03:44.769984 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.769898    2564 scope.go:117] "RemoveContainer" containerID="2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"
Jul 10 21:03:44.770295 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:03:44.770274    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63\": container with ID starting with 2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63 not found: ID does not exist" containerID="2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"
Jul 10 21:03:44.770410 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.770309    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63"} err="failed to get container status \"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63\": rpc error: code = NotFound desc = could not find container \"2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63\": container with ID starting with 2edf11892a7418b708df45384358a6e3b051d0c296e9a3b9adcdb783ee944f63 not found: ID does not exist"
Jul 10 21:03:44.770410 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.770330    2564 scope.go:117] "RemoveContainer" containerID="298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"
Jul 10 21:03:44.778616 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.778580    2564 scope.go:117] "RemoveContainer" containerID="298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"
Jul 10 21:03:44.778868 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:03:44.778850    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1\": container with ID starting with 298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1 not found: ID does not exist" containerID="298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"
Jul 10 21:03:44.778924 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.778875    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1"} err="failed to get container status \"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1\": rpc error: code = NotFound desc = could not find container \"298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1\": container with ID starting with 298389aa326385f54a8f6ae5f84a9324935cc3d67f08bfb112aff71f5f22cdb1 not found: ID does not exist"
Jul 10 21:03:44.785031 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.785013    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:03:44.788236 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.788212    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-e560d0/rate-test-model-e560d0-kserve-6d9457867f-sgn8p"]
Jul 10 21:03:44.801381 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.801358    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:03:44.805834 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:44.805811    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-28db1d/rate-test-model-28db1d-kserve-7987b9c8dc-nhzjc"]
Jul 10 21:03:46.783932 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:46.783903    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f46b02b-65b9-4c64-b93d-520508225087" path="/var/lib/kubelet/pods/8f46b02b-65b9-4c64-b93d-520508225087/volumes"
Jul 10 21:03:46.784298 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:46.784279    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" path="/var/lib/kubelet/pods/d3c4b5eb-8927-490d-b0cb-857fdccd5867/volumes"
Jul 10 21:03:48.921221 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:48.921186    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 21:03:48.921786 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:48.921751    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" podUID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" containerName="maas-api" containerID="cri-o://1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343" gracePeriod=30
Jul 10 21:03:49.172452 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.172385    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 21:03:49.328825 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.328792    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls\") pod \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") "
Jul 10 21:03:49.328966 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.328858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca\") pod \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") "
Jul 10 21:03:49.329016 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.328957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqdhf\" (UniqueName: \"kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf\") pod \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\" (UID: \"7038d7b9-eecf-4ac3-a93c-ad078207bf07\") "
Jul 10 21:03:49.329177 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.329156    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7038d7b9-eecf-4ac3-a93c-ad078207bf07" (UID: "7038d7b9-eecf-4ac3-a93c-ad078207bf07"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 21:03:49.330869 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.330847    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7038d7b9-eecf-4ac3-a93c-ad078207bf07" (UID: "7038d7b9-eecf-4ac3-a93c-ad078207bf07"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:03:49.330988 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.330955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf" (OuterVolumeSpecName: "kube-api-access-fqdhf") pod "7038d7b9-eecf-4ac3-a93c-ad078207bf07" (UID: "7038d7b9-eecf-4ac3-a93c-ad078207bf07"). InnerVolumeSpecName "kube-api-access-fqdhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:03:49.430112 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.430049    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdhf\" (UniqueName: \"kubernetes.io/projected/7038d7b9-eecf-4ac3-a93c-ad078207bf07-kube-api-access-fqdhf\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:49.430112 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.430075    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7038d7b9-eecf-4ac3-a93c-ad078207bf07-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:49.430112 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.430084    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7038d7b9-eecf-4ac3-a93c-ad078207bf07-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:49.789650 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.789570    2564 generic.go:356] "Generic (PLEG): container finished" podID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" containerID="1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343" exitCode=0
Jul 10 21:03:49.789650 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.789612    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"
Jul 10 21:03:49.789837 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.789654    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" event={"ID":"7038d7b9-eecf-4ac3-a93c-ad078207bf07","Type":"ContainerDied","Data":"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"}
Jul 10 21:03:49.789837 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.789709    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll" event={"ID":"7038d7b9-eecf-4ac3-a93c-ad078207bf07","Type":"ContainerDied","Data":"4a4cd02a99bc0b953fbca25f3f3beaeb7ac6aba411c074e700a6b575081bdadd"}
Jul 10 21:03:49.789837 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.789737    2564 scope.go:117] "RemoveContainer" containerID="1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"
Jul 10 21:03:49.798677 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.798659    2564 scope.go:117] "RemoveContainer" containerID="1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"
Jul 10 21:03:49.798957 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:03:49.798939    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343\": container with ID starting with 1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343 not found: ID does not exist" containerID="1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"
Jul 10 21:03:49.799012 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.798963    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343"} err="failed to get container status \"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343\": rpc error: code = NotFound desc = could not find container \"1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343\": container with ID starting with 1463bcba923c3911c8ffcaa7b7db99306bc2eccda78eeaf326d80030f7412343 not found: ID does not exist"
Jul 10 21:03:49.811071 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.811046    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 21:03:49.814549 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:49.814530    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-e560d0-d4784bbc-5gdll"]
Jul 10 21:03:50.784509 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:50.784458    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" path="/var/lib/kubelet/pods/7038d7b9-eecf-4ac3-a93c-ad078207bf07/volumes"
Jul 10 21:03:51.929972 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:51.929939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:03:56.177550 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.177519    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 21:03:56.178012 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.177795    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" podUID="762bcf66-74ff-4637-9b78-a84c4184f446" containerName="maas-api" containerID="cri-o://011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597" gracePeriod=30
Jul 10 21:03:56.633101 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.633077    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 21:03:56.682788 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.682762    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q747\" (UniqueName: \"kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747\") pod \"762bcf66-74ff-4637-9b78-a84c4184f446\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") "
Jul 10 21:03:56.682965 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.682797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca\") pod \"762bcf66-74ff-4637-9b78-a84c4184f446\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") "
Jul 10 21:03:56.683160 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.683134    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "762bcf66-74ff-4637-9b78-a84c4184f446" (UID: "762bcf66-74ff-4637-9b78-a84c4184f446"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 10 21:03:56.684825 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.684803    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747" (OuterVolumeSpecName: "kube-api-access-5q747") pod "762bcf66-74ff-4637-9b78-a84c4184f446" (UID: "762bcf66-74ff-4637-9b78-a84c4184f446"). InnerVolumeSpecName "kube-api-access-5q747". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 10 21:03:56.783612 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.783534    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") pod \"762bcf66-74ff-4637-9b78-a84c4184f446\" (UID: \"762bcf66-74ff-4637-9b78-a84c4184f446\") "
Jul 10 21:03:56.783799 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.783785    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q747\" (UniqueName: \"kubernetes.io/projected/762bcf66-74ff-4637-9b78-a84c4184f446-kube-api-access-5q747\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:56.783850 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.783806    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/762bcf66-74ff-4637-9b78-a84c4184f446-openshift-service-ca\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:56.785539 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.785516    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "762bcf66-74ff-4637-9b78-a84c4184f446" (UID: "762bcf66-74ff-4637-9b78-a84c4184f446"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 10 21:03:56.822883 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.822855    2564 generic.go:356] "Generic (PLEG): container finished" podID="762bcf66-74ff-4637-9b78-a84c4184f446" containerID="011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597" exitCode=0
Jul 10 21:03:56.822993 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.822903    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"
Jul 10 21:03:56.822993 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.822929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" event={"ID":"762bcf66-74ff-4637-9b78-a84c4184f446","Type":"ContainerDied","Data":"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"}
Jul 10 21:03:56.822993 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.822967    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb" event={"ID":"762bcf66-74ff-4637-9b78-a84c4184f446","Type":"ContainerDied","Data":"45cf6efffc56b571c48e6116851a7ba60b7b6022ca2d25cc2b5b3db9b3468aac"}
Jul 10 21:03:56.822993 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.822990    2564 scope.go:117] "RemoveContainer" containerID="011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"
Jul 10 21:03:56.831154 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.831137    2564 scope.go:117] "RemoveContainer" containerID="011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"
Jul 10 21:03:56.831443 ip-10-0-141-240 kubenswrapper[2564]: E0710 21:03:56.831419    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597\": container with ID starting with 011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597 not found: ID does not exist" containerID="011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"
Jul 10 21:03:56.831509 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.831444    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597"} err="failed to get container status \"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597\": rpc error: code = NotFound desc = could not find container \"011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597\": container with ID starting with 011f52ee6d044a58e4f946969a73f043f569ab286eec538f86cf9de43036b597 not found: ID does not exist"
Jul 10 21:03:56.839577 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.839550    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 21:03:56.841609 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.841586    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-28db1d-5587c4777c-c5vqb"]
Jul 10 21:03:56.884715 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:56.884693    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/762bcf66-74ff-4637-9b78-a84c4184f446-maas-api-tls\") on node \"ip-10-0-141-240.ec2.internal\" DevicePath \"\""
Jul 10 21:03:58.784931 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:03:58.784899    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762bcf66-74ff-4637-9b78-a84c4184f446" path="/var/lib/kubelet/pods/762bcf66-74ff-4637-9b78-a84c4184f446/volumes"
Jul 10 21:04:05.032283 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:05.032241    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-ngphh"]
Jul 10 21:04:07.078251 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.078212    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ktsqc_c0b1fa2a-544b-4182-98a2-0bd8be73b7b3/manager/0.log"
Jul 10 21:04:07.196231 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.196199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5vhhr_20dacbce-46d6-4a6e-97c4-3e7b97b069c9/manager/0.log"
Jul 10 21:04:07.317319 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.317284    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6789fdfff6-kxdbj_540c12c3-dec8-468b-8cad-16b1da9bedea/manager/0.log"
Jul 10 21:04:07.433734 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.433654    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7j4hq_b7346fff-038b-4b2d-9366-b09e2e129acf/server/0.log"
Jul 10 21:04:07.551586 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.551560    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9fg6_81c51fea-0221-4fe7-8438-1a6523469630/manager/0.log"
Jul 10 21:04:07.662543 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.662516    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6f977c45-5x8q2_4231385e-84e6-4f1d-8661-37860eb81dea/copy-manifests/0.log"
Jul 10 21:04:07.675593 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:07.675566    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6f977c45-5x8q2_4231385e-84e6-4f1d-8661-37860eb81dea/manager/0.log"
Jul 10 21:04:08.269008 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:08.268969    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-84fc6df8d5-k2j6v_506bf005-d114-4857-aeb9-0c7f34aad58c/maas-api/0.log"
Jul 10 21:04:08.391252 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:08.391216    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-24sds_5ba88514-3230-4b67-a8f3-24c53e7127f6/postgres/0.log"
Jul 10 21:04:09.749944 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:09.749912    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h9lj5_4f83aa55-b54c-4057-9218-6f897706ff1d/manager/0.log"
Jul 10 21:04:09.862961 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:09.862931    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s77sx_f2a308e0-f01f-47a8-9015-783d08471b7f/manager/0.log"
Jul 10 21:04:09.964719 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:09.964691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kfw5w_457920d3-7eac-4752-ab7b-f9d389f983b2/kuadrant-console-plugin/0.log"
Jul 10 21:04:10.077916 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:10.077881    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jvst7_8455dd04-f0e8-4379-98b4-c09e31b37677/registry-server/0.log"
Jul 10 21:04:10.293271 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:10.293246    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ngphh_3d50bb32-67fb-4525-bad7-7f2f9b32549c/limitador/0.log"
Jul 10 21:04:10.407220 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:10.407139    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-th8zc_902aac0c-29f6-4f7f-a53c-fffc6a69ce25/manager/0.log"
Jul 10 21:04:11.075718 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:11.075675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5fcf8fc5-kj45j_a8f9f0e5-c3d1-43d7-ac43-acf8b297107e/kube-auth-proxy/0.log"
Jul 10 21:04:11.957126 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:11.957088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dln2j_4c6ed507-5e7d-49a4-a3e9-fc3c533da0e9/main/0.log"
Jul 10 21:04:12.283882 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:12.283802    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ddshx_4470944b-fc16-47a4-b856-778e2de25dbd/main/0.log"
Jul 10 21:04:19.230912 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:19.230876    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gqc7d_8095de59-42e6-4f38-9280-7761a61ef4ae/global-pull-secret-syncer/0.log"
Jul 10 21:04:19.415673 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:19.415639    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-r246c_a6681f43-e63d-495d-9870-62d20624cac7/konnectivity-agent/0.log"
Jul 10 21:04:19.494302 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:19.494207    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-240.ec2.internal_869786661d939b62b49e72c2e18cdc59/haproxy/0.log"
Jul 10 21:04:24.732944 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:24.732901    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-h9lj5_4f83aa55-b54c-4057-9218-6f897706ff1d/manager/0.log"
Jul 10 21:04:24.755035 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:24.755009    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-s77sx_f2a308e0-f01f-47a8-9015-783d08471b7f/manager/0.log"
Jul 10 21:04:24.783803 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:24.783779    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-kfw5w_457920d3-7eac-4752-ab7b-f9d389f983b2/kuadrant-console-plugin/0.log"
Jul 10 21:04:24.830260 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:24.830233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jvst7_8455dd04-f0e8-4379-98b4-c09e31b37677/registry-server/0.log"
Jul 10 21:04:25.045100 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:25.045016    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-ngphh_3d50bb32-67fb-4525-bad7-7f2f9b32549c/limitador/0.log"
Jul 10 21:04:25.139391 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:25.139354    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-th8zc_902aac0c-29f6-4f7f-a53c-fffc6a69ce25/manager/0.log"
Jul 10 21:04:27.291563 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:27.291533    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8gxp_8d545587-46d7-4c0b-b6d5-48054c46debe/node-exporter/0.log"
Jul 10 21:04:27.310662 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:27.310635    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8gxp_8d545587-46d7-4c0b-b6d5-48054c46debe/kube-rbac-proxy/0.log"
Jul 10 21:04:27.330874 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:27.330852    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p8gxp_8d545587-46d7-4c0b-b6d5-48054c46debe/init-textfile/0.log"
Jul 10 21:04:30.029147 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029113    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"]
Jul 10 21:04:30.029575 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029556    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" containerName="maas-api"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029577    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" containerName="maas-api"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029585    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" containerName="main"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029591    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" containerName="main"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029609    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="762bcf66-74ff-4637-9b78-a84c4184f446" containerName="maas-api"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029614    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="762bcf66-74ff-4637-9b78-a84c4184f446" containerName="maas-api"
Jul 10 21:04:30.029626 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029625    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f46b02b-65b9-4c64-b93d-520508225087" containerName="main"
Jul 10 21:04:30.029832 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029630    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f46b02b-65b9-4c64-b93d-520508225087" containerName="main"
Jul 10 21:04:30.029832 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029694    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7038d7b9-eecf-4ac3-a93c-ad078207bf07" containerName="maas-api"
Jul 10 21:04:30.029832 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029704    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f46b02b-65b9-4c64-b93d-520508225087" containerName="main"
Jul 10 21:04:30.029832 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029714    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="762bcf66-74ff-4637-9b78-a84c4184f446" containerName="maas-api"
Jul 10 21:04:30.029832 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.029723    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c4b5eb-8927-490d-b0cb-857fdccd5867" containerName="main"
Jul 10 21:04:30.032483 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.032462    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.035559 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.035540    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n2z9x\"/\"kube-root-ca.crt\""
Jul 10 21:04:30.036800 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.036775    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n2z9x\"/\"default-dockercfg-kpjj7\""
Jul 10 21:04:30.036915 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.036799    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n2z9x\"/\"openshift-service-ca.crt\""
Jul 10 21:04:30.043996 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.043974    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"]
Jul 10 21:04:30.145331 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.145288    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-sys\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.145331 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.145326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-proc\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.145593 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.145358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-lib-modules\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.145593 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.145382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-podres\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.145593 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.145400    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthwx\" (UniqueName: \"kubernetes.io/projected/7293bdd4-4c13-478e-8d41-1e6b28c61c64-kube-api-access-zthwx\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246170 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-sys\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246170 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-proc\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246427 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246264    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-proc\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246427 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246267    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-sys\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246427 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-lib-modules\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246427 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246340    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-podres\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246427 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthwx\" (UniqueName: \"kubernetes.io/projected/7293bdd4-4c13-478e-8d41-1e6b28c61c64-kube-api-access-zthwx\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246675 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-lib-modules\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.246675 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.246537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7293bdd4-4c13-478e-8d41-1e6b28c61c64-podres\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.255065 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.255046    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthwx\" (UniqueName: \"kubernetes.io/projected/7293bdd4-4c13-478e-8d41-1e6b28c61c64-kube-api-access-zthwx\") pod \"perf-node-gather-daemonset-7mplb\" (UID: \"7293bdd4-4c13-478e-8d41-1e6b28c61c64\") " pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.342425 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.342402    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.474301 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.474275    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"]
Jul 10 21:04:30.477098 ip-10-0-141-240 kubenswrapper[2564]: W0710 21:04:30.477072    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7293bdd4_4c13_478e_8d41_1e6b28c61c64.slice/crio-6c1f15fda815cdfc661162fb7b6a33dd2fa49e747778a38eef1ef009a26f2520 WatchSource:0}: Error finding container 6c1f15fda815cdfc661162fb7b6a33dd2fa49e747778a38eef1ef009a26f2520: Status 404 returned error can't find the container with id 6c1f15fda815cdfc661162fb7b6a33dd2fa49e747778a38eef1ef009a26f2520
Jul 10 21:04:30.976934 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.976901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb" event={"ID":"7293bdd4-4c13-478e-8d41-1e6b28c61c64","Type":"ContainerStarted","Data":"42e6848cedc6d2e44888e2ce1a8d40f69fe661c75ee59c354f62e992db16f282"}
Jul 10 21:04:30.977107 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.976951    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb" event={"ID":"7293bdd4-4c13-478e-8d41-1e6b28c61c64","Type":"ContainerStarted","Data":"6c1f15fda815cdfc661162fb7b6a33dd2fa49e747778a38eef1ef009a26f2520"}
Jul 10 21:04:30.977107 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.976981    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:30.999204 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:30.994819    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb" podStartSLOduration=0.994800756 podStartE2EDuration="994.800756ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 21:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 21:04:30.992995862 +0000 UTC m=+2594.893301083" watchObservedRunningTime="2026-07-10 21:04:30.994800756 +0000 UTC m=+2594.895105975"
Jul 10 21:04:32.056979 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:32.056949    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nc87d_58873ee7-32c0-4d3b-a18a-3ab6fd149297/dns/0.log"
Jul 10 21:04:32.087597 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:32.087565    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nc87d_58873ee7-32c0-4d3b-a18a-3ab6fd149297/kube-rbac-proxy/0.log"
Jul 10 21:04:32.274748 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:32.274714    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hjqwh_395ceee6-e47f-4442-8897-6b4a0075f68b/dns-node-resolver/0.log"
Jul 10 21:04:32.866072 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:32.866043    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnq7j_b5d0e425-31ca-4ab8-bd99-8534f583d901/node-ca/0.log"
Jul 10 21:04:34.004158 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:34.004127    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5fcf8fc5-kj45j_a8f9f0e5-c3d1-43d7-ac43-acf8b297107e/kube-auth-proxy/0.log"
Jul 10 21:04:34.866080 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:34.866029    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r2j9g_aa14fe47-a4ae-4b17-8b81-3965e3c2674b/serve-healthcheck-canary/0.log"
Jul 10 21:04:35.369327 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:35.369301    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-645jh_ea90649f-2f85-417d-afde-75123e21ad70/kube-rbac-proxy/0.log"
Jul 10 21:04:35.391123 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:35.391095    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-645jh_ea90649f-2f85-417d-afde-75123e21ad70/exporter/0.log"
Jul 10 21:04:35.412676 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:35.412644    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-645jh_ea90649f-2f85-417d-afde-75123e21ad70/extractor/0.log"
Jul 10 21:04:36.992406 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:36.992382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n2z9x/perf-node-gather-daemonset-7mplb"
Jul 10 21:04:37.782705 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:37.782675    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ktsqc_c0b1fa2a-544b-4182-98a2-0bd8be73b7b3/manager/0.log"
Jul 10 21:04:38.011631 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.011594    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-5vhhr_20dacbce-46d6-4a6e-97c4-3e7b97b069c9/manager/0.log"
Jul 10 21:04:38.163257 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.163220    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6789fdfff6-kxdbj_540c12c3-dec8-468b-8cad-16b1da9bedea/manager/0.log"
Jul 10 21:04:38.183670 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.183640    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-7j4hq_b7346fff-038b-4b2d-9366-b09e2e129acf/server/0.log"
Jul 10 21:04:38.284360 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.284285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n9fg6_81c51fea-0221-4fe7-8438-1a6523469630/manager/0.log"
Jul 10 21:04:38.316672 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.316646    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6f977c45-5x8q2_4231385e-84e6-4f1d-8661-37860eb81dea/manager/0.log"
Jul 10 21:04:38.335786 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:38.335761    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b6f977c45-5x8q2_4231385e-84e6-4f1d-8661-37860eb81dea/copy-manifests/0.log"
Jul 10 21:04:40.039758 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:40.039731    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-snhzk_f7745869-52e5-45bf-abbd-bf751dd3174f/openshift-lws-operator/0.log"
Jul 10 21:04:40.152761 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:40.152733    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5fcf8fc5-kj45j_a8f9f0e5-c3d1-43d7-ac43-acf8b297107e/kube-auth-proxy/0.log"
Jul 10 21:04:41.567783 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:41.567755    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5fcf8fc5-kj45j_a8f9f0e5-c3d1-43d7-ac43-acf8b297107e/kube-auth-proxy/0.log"
Jul 10 21:04:46.413687 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.413658    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4cnx8_d8317e36-4f62-40a7-91cd-0b523511bbb2/kube-multus/0.log"
Jul 10 21:04:46.438951 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.438930    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/kube-multus-additional-cni-plugins/0.log"
Jul 10 21:04:46.462342 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.462308    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/egress-router-binary-copy/0.log"
Jul 10 21:04:46.483253 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.483229    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/cni-plugins/0.log"
Jul 10 21:04:46.504682 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.504660    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/bond-cni-plugin/0.log"
Jul 10 21:04:46.527941 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.527913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/routeoverride-cni/0.log"
Jul 10 21:04:46.549027 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.549007    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/whereabouts-cni-bincopy/0.log"
Jul 10 21:04:46.571641 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:46.571623    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6ww8b_74670a37-c1b7-4727-9de0-8281ca2176c1/whereabouts-cni/0.log"
Jul 10 21:04:47.032753 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:47.032720    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cfp74_2b72c49e-6395-4d9f-9495-d52457c4680c/network-metrics-daemon/0.log"
Jul 10 21:04:47.052404 ip-10-0-141-240 kubenswrapper[2564]: I0710 21:04:47.052386    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cfp74_2b72c49e-6395-4d9f-9495-d52457c4680c/kube-rbac-proxy/0.log"