Jul 22 18:14:05.670069 ip-10-0-141-65 systemd[1]: Starting Kubernetes Kubelet...
Jul 22 18:14:06.044254 ip-10-0-141-65 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 18:14:06.044254 ip-10-0-141-65 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 22 18:14:06.044254 ip-10-0-141-65 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 18:14:06.044254 ip-10-0-141-65 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 22 18:14:06.044254 ip-10-0-141-65 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 22 18:14:06.045281 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.045200    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 22 18:14:06.049083 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049067    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.049083 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049082    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.049083 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049086    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049089    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049092    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049095    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049098    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049101    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049104    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049107    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049109    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049112    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049121    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049124    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049126    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049129    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049132    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049134    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049137    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049139    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049142    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049145    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.049177 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049147    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049150    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049152    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049155    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049158    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049160    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049164    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049168    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049171    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049178    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049181    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049184    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049188    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049191    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049194    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049197    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049199    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049201    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049204    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.049638 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049206    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049209    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049211    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049214    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049216    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049218    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049221    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049223    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049226    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049228    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049230    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049233    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049235    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049238    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049240    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049245    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049248    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049250    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049252    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049255    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.050116 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049257    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049260    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049262    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049265    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049268    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049271    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049273    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049276    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049278    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049280    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049283    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049286    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049289    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049291    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049294    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049297    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049300    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049302    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049305    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049308    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.050566 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049311    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049315    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049319    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049323    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049326    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049329    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049332    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049335    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049338    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049341    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049344    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049346    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049349    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049351    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049354    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049357    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049360    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.049363    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050112    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.051062 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050119    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050123    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050126    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050130    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050134    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050137    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050139    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050142    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050145    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050148    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050151    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050154    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050157    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050160    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050163    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050166    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050168    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050170    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050173    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050175    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.051503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050178    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050182    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050186    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050189    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050192    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050195    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050197    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050200    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050202    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050205    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050208    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050210    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050213    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050215    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050217    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050220    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050223    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050226    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050228    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.052020 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050231    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050233    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050237    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050239    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050241    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050244    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050247    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050250    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050252    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050255    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050257    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050260    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050263    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050266    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050268    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050271    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050273    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050276    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050278    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050281    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.052473 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050284    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050286    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050289    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050291    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050297    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050300    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050302    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050305    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050307    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050310    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050312    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050315    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050318    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050321    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050323    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050326    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050328    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050331    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050335    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050339    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.052949 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050341    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050344    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050346    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050349    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050352    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050354    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050357    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050359    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050362    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050364    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050368    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050374    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050377    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050379    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050382    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050385    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050387    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050397    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050400    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.053419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050775    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050780    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050785    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050788    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050791    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050795    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050798    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050800    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050818    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050821    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050824    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050826    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050829    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050832    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050834    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050837    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050839    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050842    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050845    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.053905 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050847    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050850    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050852    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050855    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050858    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050863    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050865    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050868    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050872    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050875    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050878    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050884    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050888    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050891    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050894    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050897    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050899    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050902    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050905    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050907    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.054363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050909    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050912    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050914    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050917    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050919    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050922    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050924    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050927    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050929    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050932    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050934    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050936    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050939    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050941    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050944    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050946    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050949    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050953    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050956    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050959    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.054862 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050961    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050964    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050967    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050969    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050972    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050975    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050978    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050982    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050985    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050987    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050991    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050993    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050996    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.050998    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051001    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051003    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051006    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051008    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051011    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051013    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.055368 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051016    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051018    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051021    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051024    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051026    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051029    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051031    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051034    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051036    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051040    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051043    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051045    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051048    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051050    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051053    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051055    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051058    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051060    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051063    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.055878 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051066    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051135    2580 flags.go:64] FLAG: --address="0.0.0.0"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051146    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051151    2580 flags.go:64] FLAG: --anonymous-auth="true"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051155    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051160    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051163    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051167    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051171    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051174    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051189    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051194    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051198    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051202    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051205    2580 flags.go:64] FLAG: --cgroup-root=""
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051208    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051210    2580 flags.go:64] FLAG: --client-ca-file=""
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051213    2580 flags.go:64] FLAG: --cloud-provider="external"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051216    2580 flags.go:64] FLAG: --cluster-dns="[]"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051222    2580 flags.go:64] FLAG: --cluster-domain=""
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051224    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051227    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051235    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051238    2580 flags.go:64] FLAG: --container-log-max-files="5"
Jul 22 18:14:06.056324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051242    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051245    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051248    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051252    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051255    2580 flags.go:64] FLAG: --contention-profiling="false"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051258    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051261    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051264    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051267    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051271    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051274    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051277    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051280    2580 flags.go:64] FLAG: --enable-load-reader="false"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051282    2580 flags.go:64] FLAG: --enable-server="true"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051286    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051290    2580 flags.go:64] FLAG: --event-burst="100"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051293    2580 flags.go:64] FLAG: --event-qps="50"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051296    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051299    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051302    2580 flags.go:64] FLAG: --eviction-hard=""
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051306    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051309    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051312    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051315    2580 flags.go:64] FLAG: --eviction-soft=""
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051318    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 22 18:14:06.056922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051321    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051323    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051326    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051330    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051332    2580 flags.go:64] FLAG: --fail-swap-on="true"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051338    2580 flags.go:64] FLAG: --feature-gates=""
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051341    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051345    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051348    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051351    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051354    2580 flags.go:64] FLAG: --healthz-port="10248"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051357    2580 flags.go:64] FLAG: --help="false"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051359    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051363    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051366    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051369    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051372    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051376    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051379    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051382    2580 flags.go:64] FLAG: --image-service-endpoint=""
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051384    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051387    2580 flags.go:64] FLAG: --kube-api-burst="100"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051390    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051393    2580 flags.go:64] FLAG: --kube-api-qps="50"
Jul 22 18:14:06.057515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051396    2580 flags.go:64] FLAG: --kube-reserved=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051399    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051401    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051404    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051407    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051410    2580 flags.go:64] FLAG: --lock-file=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051412    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051415    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051418    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051423    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051426    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051429    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051432    2580 flags.go:64] FLAG: --logging-format="text"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051437    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051440    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051443    2580 flags.go:64] FLAG: --manifest-url=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051446    2580 flags.go:64] FLAG: --manifest-url-header=""
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051450    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051453    2580 flags.go:64] FLAG: --max-open-files="1000000"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051457    2580 flags.go:64] FLAG: --max-pods="110"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051460    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051463    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051466    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051469    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051473    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 22 18:14:06.058142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051476    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051479    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051485    2580 flags.go:64] FLAG: --node-status-max-images="50"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051488    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051491    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051494    2580 flags.go:64] FLAG: --pod-cidr=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051497    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051503    2580 flags.go:64] FLAG: --pod-manifest-path=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051506    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051509    2580 flags.go:64] FLAG: --pods-per-core="0"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051512    2580 flags.go:64] FLAG: --port="10250"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051515    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051519    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03b9e7529a7e223b4"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051522    2580 flags.go:64] FLAG: --qos-reserved=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051525    2580 flags.go:64] FLAG: --read-only-port="10255"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051528    2580 flags.go:64] FLAG: --register-node="true"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051531    2580 flags.go:64] FLAG: --register-with-taints=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051534    2580 flags.go:64] FLAG: --registry-burst="10"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051537    2580 flags.go:64] FLAG: --registry-qps="5"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051540    2580 flags.go:64] FLAG: --reserved-cpus=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051543    2580 flags.go:64] FLAG: --reserved-memory=""
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051548    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051551    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051554    2580 flags.go:64] FLAG: --rotate-certificates="false"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051557    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 22 18:14:06.058739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051560    2580 flags.go:64] FLAG: --runonce="false"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051563    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051566    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051569    2580 flags.go:64] FLAG: --seccomp-default="false"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051572    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051575    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051578    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051581    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051584    2580 flags.go:64] FLAG: --storage-driver-password="root"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051587    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051590    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051593    2580 flags.go:64] FLAG: --storage-driver-user="root"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051596    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051599    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051602    2580 flags.go:64] FLAG: --system-cgroups=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051604    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051610    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051612    2580 flags.go:64] FLAG: --tls-cert-file=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051615    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051619    2580 flags.go:64] FLAG: --tls-min-version=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051622    2580 flags.go:64] FLAG: --tls-private-key-file=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051625    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051628    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051631    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051634    2580 flags.go:64] FLAG: --v="2"
Jul 22 18:14:06.059426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051638    2580 flags.go:64] FLAG: --version="false"
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051642    2580 flags.go:64] FLAG: --vmodule=""
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051645    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.051651    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051772    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051776    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051779    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051782    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051785    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051788    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051791    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051794    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051796    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051799    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051801    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051817    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051820    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051823    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051826    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051829    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.060094 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051831    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051834    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051836    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051839    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051842    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051844    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051847    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051849    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051852    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051854    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051857    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051859    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051861    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051864    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051866    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051871    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051873    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051876    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051878    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051881    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.060609 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051883    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051886    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051889    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051892    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051894    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051897    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051900    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051902    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051906    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051908    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051911    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051913    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051916    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051918    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051921    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051923    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051926    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051930    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051934    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.061379 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051937    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051940    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051942    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051945    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051948    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051950    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051953    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051957    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051961    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051964    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051967    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051970    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051972    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051975    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051977    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051980    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051983    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051985    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051988    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051990    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.061891 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051993    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051995    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.051998    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052004    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052007    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052010    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052013    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052016    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052019    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052022    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052024    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052027    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052030    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052033    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052036    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052038    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052041    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052044    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052046    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.062503 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052049    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052052    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052055    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052058    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.052060    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.052816    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.059288    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.059302    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.060022    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.060031    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060090    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060096    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060099    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060103    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060106    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.062991 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060110    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060113    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060116    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060119    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060122    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060124    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060127    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060129    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060132    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060134    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060137    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060140    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060142    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060145    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060147    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060150    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060152    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060155    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060157    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060160    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.063371 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060162    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060165    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060168    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060171    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060174    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060176    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060179    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060182    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060184    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060187    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060189    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060193    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060195    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060198    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060201    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060203    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060206    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060208    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060211    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060214    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.063880 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060217    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060219    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060222    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060224    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060227    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060229    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060231    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060234    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060236    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060239    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060241    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060244    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060246    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060249    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060252    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060254    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060257    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060259    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060262    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060265    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.064347 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060267    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060270    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060272    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060275    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060279    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060282    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060286    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060289    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060291    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060294    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060296    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060299    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060303    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060306    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060309    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060312    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060315    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060317    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060320    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.064835 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060323    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060325    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060327    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060330    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060333    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060335    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060338    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060341    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060343    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060346    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060349    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060351    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060355    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060358    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.065338 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060361    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.060367    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060582    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060590    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060593    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060596    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060609    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060612    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060620    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060624    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.060629    2580 feature_gate.go:328] unrecognized feature gate: Example
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062126    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062147    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062153    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062162    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 22 18:14:06.065678 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062168    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062175    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062181    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062186    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062193    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062201    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062207    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062213    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062224    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062231    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062237    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062242    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062247    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062279    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062289    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062294    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062298    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062302    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062320    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 22 18:14:06.066155 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062346    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062458    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062468    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062472    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062476    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062480    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062485    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062489    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062493    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062497    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062501    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062506    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062510    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062514    2580 feature_gate.go:328] unrecognized feature gate: Example2
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062519    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062523    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062527    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062531    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062535    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062539    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 22 18:14:06.066674 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062546    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062552    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062556    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062560    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062564    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062568    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062572    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062576    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062580    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062585    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062589    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062593    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062598    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062602    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062607    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062611    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062615    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062620    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062624    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062628    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 22 18:14:06.067257 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062633    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062636    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062642    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062646    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062650    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062654    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062659    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062663    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062667    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062671    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062675    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062680    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062684    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062686    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062689    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062692    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062698    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062702    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062706    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062709    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 22 18:14:06.067752 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062712    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062715    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062718    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062722    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062726    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062729    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:06.062732    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.062738    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.062875    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.064901    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.066154    2580 server.go:1019] "Starting client certificate rotation"
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.066279    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 18:14:06.068333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.066965    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 22 18:14:06.068624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.068575    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 18:14:06.069760 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.069731    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 22 18:14:06.082915 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.082893    2580 log.go:25] "Validated CRI v1 runtime API"
Jul 22 18:14:06.085655 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.085642    2580 log.go:25] "Validated CRI v1 image API"
Jul 22 18:14:06.086453 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.086434    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 22 18:14:06.090833 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.090726    2580 fs.go:135] Filesystem UUIDs: map[4bdbcd8b-9c6b-4fce-84ef-295dc8a0c600:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd23cf0d-d831-4876-9633-9f40157d43ae:/dev/nvme0n1p4]
Jul 22 18:14:06.090833 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.090748    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 22 18:14:06.095236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.095213    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 18:14:06.096702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.096594    2580 manager.go:216] Machine: {Timestamp:2026-07-22 18:14:06.095390783 +0000 UTC m=+0.311038178 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098677 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca224c758924ab5b59b6ae2ce4cf9 SystemUUID:ec2ca224-c758-924a-b5b5-9b6ae2ce4cf9 BootID:ee5f1e87-63b0-4fd0-b537-5a1fa8fbe5b3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d8:3f:e9:70:81 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d8:3f:e9:70:81 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:b0:68:54:85:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 22 18:14:06.096702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.096695    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 22 18:14:06.096884 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.096861    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 22 18:14:06.099446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.099419    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 22 18:14:06.099601 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.099449    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 22 18:14:06.100683 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.100671    2580 topology_manager.go:138] "Creating topology manager with none policy"
Jul 22 18:14:06.100740 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.100689    2580 container_manager_linux.go:309] "Creating device plugin manager"
Jul 22 18:14:06.100740 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.100700    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 18:14:06.101362 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.101351    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 22 18:14:06.102592 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.102580    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 22 18:14:06.105266 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105253    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 18:14:06.105409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105398    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 22 18:14:06.105474 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105433    2580 kubelet.go:486] "Attempting to sync node with API server"
Jul 22 18:14:06.105474 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105447    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 22 18:14:06.105474 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105466    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 22 18:14:06.105595 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105478    2580 kubelet.go:398] "Adding apiserver pod source"
Jul 22 18:14:06.105595 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105490    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 22 18:14:06.105882 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105869    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 18:14:06.105944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.105892    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 22 18:14:06.108899 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.108866    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 22 18:14:06.110944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.110929    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 22 18:14:06.111016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.110961    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 22 18:14:06.112069 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112054    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112073    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112080    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112089    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112099    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112105    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112113    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112118    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 22 18:14:06.112124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112125    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 22 18:14:06.112343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112131    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 22 18:14:06.112343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112141    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 22 18:14:06.112343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112148    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 22 18:14:06.112841 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112831    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 22 18:14:06.112841 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.112841    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 22 18:14:06.114301 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.114274    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 22 18:14:06.114301 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.114289    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 22 18:14:06.116201 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.116188    2580 server.go:1268] "Started kubelet"
Jul 22 18:14:06.116331 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.116287    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 22 18:14:06.116368 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.116347    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 22 18:14:06.116733 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.116708    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 22 18:14:06.116980 ip-10-0-141-65 systemd[1]: Started Kubernetes Kubelet.
Jul 22 18:14:06.118047 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.117921    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 22 18:14:06.118792 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.118776    2580 server.go:310] "Adding debug handlers to kubelet server"
Jul 22 18:14:06.121649 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.121627    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tj6l"
Jul 22 18:14:06.121711 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.121660    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 22 18:14:06.123906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.123890    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 22 18:14:06.123906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.123901    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 22 18:14:06.124006 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.123953    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.124509 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124491    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 22 18:14:06.124623 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124608    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 22 18:14:06.124702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124694    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 22 18:14:06.124781 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124640    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 22 18:14:06.124896 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124882    2580 reconstruct.go:97] "Volume reconstruction finished"
Jul 22 18:14:06.124950 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.124896    2580 reconciler.go:29] "Reconciler: start to sync state"
Jul 22 18:14:06.125039 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125021    2580 factory.go:55] Registering systemd factory
Jul 22 18:14:06.125039 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125039    2580 factory.go:223] Registration of the systemd container factory successfully
Jul 22 18:14:06.125272 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125257    2580 factory.go:153] Registering CRI-O factory
Jul 22 18:14:06.125320 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125277    2580 factory.go:223] Registration of the crio container factory successfully
Jul 22 18:14:06.125357 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125323    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 22 18:14:06.125357 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125344    2580 factory.go:103] Registering Raw factory
Jul 22 18:14:06.125454 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125358    2580 manager.go:1196] Started watching for new ooms in manager
Jul 22 18:14:06.125769 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.125742    2580 manager.go:349] Starting recovery of all containers
Jul 22 18:14:06.125850 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.124663    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-65.ec2.internal.18c4af4103bf505c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-65.ec2.internal,UID:ip-10-0-141-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-65.ec2.internal,},FirstTimestamp:2026-07-22 18:14:06.116163676 +0000 UTC m=+0.331811071,LastTimestamp:2026-07-22 18:14:06.116163676 +0000 UTC m=+0.331811071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-65.ec2.internal,}"
Jul 22 18:14:06.128290 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.128255    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 22 18:14:06.128385 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.128368    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 22 18:14:06.132524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.132496    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7tj6l"
Jul 22 18:14:06.137314 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.137297    2580 manager.go:354] Recovery completed
Jul 22 18:14:06.138927 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.138904    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 22 18:14:06.141793 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.141781    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.143906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.143891    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.143965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.143917    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.143965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.143928    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.144411 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.144396    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 22 18:14:06.144411 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.144408    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.144499 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.144409    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 22 18:14:06.144499 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.144477    2580 state_mem.go:36] "Initialized new in-memory state store"
Jul 22 18:14:06.145070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145055    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.145145 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145086    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.145145 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145100    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.145717 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145702    2580 policy_none.go:49] "None policy: Start"
Jul 22 18:14:06.145717 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145717    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 22 18:14:06.145858 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.145729    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 22 18:14:06.145907 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.145796    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-65.ec2.internal.18c4af4105669f2e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-65.ec2.internal,UID:ip-10-0-141-65.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-65.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-65.ec2.internal,},FirstTimestamp:2026-07-22 18:14:06.143905582 +0000 UTC m=+0.359552975,LastTimestamp:2026-07-22 18:14:06.143905582 +0000 UTC m=+0.359552975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-65.ec2.internal,}"
Jul 22 18:14:06.146768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.146757    2580 policy_none.go:47] "Start"
Jul 22 18:14:06.188345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188330    2580 manager.go:337] "Starting Device Plugin manager"
Jul 22 18:14:06.188422 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.188359    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 22 18:14:06.188422 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188369    2580 server.go:87] "Starting device plugin registration server"
Jul 22 18:14:06.188630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188618    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 22 18:14:06.188677 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188633    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 22 18:14:06.188736 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188720    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 22 18:14:06.188849 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188787    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 22 18:14:06.188849 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.188798    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 22 18:14:06.189361 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.189329    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 22 18:14:06.189447 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.189383    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.289243 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.289207    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.290708 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.290692    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.290774 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.290722    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.290774 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.290734    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.290774 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.290760    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.296641 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.296444    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.296641 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.296616    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-65.ec2.internal\": node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.312163 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.312138    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.312581 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.312558    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 22 18:14:06.313937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.313913    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 22 18:14:06.314038 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.313962    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 22 18:14:06.314038 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.313991    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 22 18:14:06.314038 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.314032    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 22 18:14:06.316320 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.316300    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 18:14:06.413866 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.413792    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.415000 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.414978    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"]
Jul 22 18:14:06.415070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.415060    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.416142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.416128    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.416196 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.416159    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.416196 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.416169    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.417587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.417575    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.417731 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.417716    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.417785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.417750    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.418321 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418299    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.418321 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418320    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.418502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418331    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.418502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418341    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.418502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418348    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.418502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.418352    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.419706 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.419686    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.419801 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.419715    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 22 18:14:06.420386 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.420371    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientMemory"
Jul 22 18:14:06.420462 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.420395    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasNoDiskPressure"
Jul 22 18:14:06.420462 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.420406    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeHasSufficientPID"
Jul 22 18:14:06.426444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.426426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14abac64ed2c3ba96221ac6ba71509b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.ec2.internal\" (UID: \"b14abac64ed2c3ba96221ac6ba71509b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.426533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.426452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.426533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.426472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.441678 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.441656    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-65.ec2.internal\" not found" node="ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.445292 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.445272    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-65.ec2.internal\" not found" node="ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.513991 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.513954    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.526627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14abac64ed2c3ba96221ac6ba71509b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.ec2.internal\" (UID: \"b14abac64ed2c3ba96221ac6ba71509b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.526763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526634    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.526763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.526763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.526763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526697    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b14abac64ed2c3ba96221ac6ba71509b-config\") pod \"kube-apiserver-proxy-ip-10-0-141-65.ec2.internal\" (UID: \"b14abac64ed2c3ba96221ac6ba71509b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.526763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.526720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d3fb81ff22ef077bddb6ace0b924312-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal\" (UID: \"0d3fb81ff22ef077bddb6ace0b924312\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.614453 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.614364    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.721055 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.721013    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.745313 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.745276    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.748179 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:06.748157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:06.823052 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.823001    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:06.930371 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:06.930290    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.030453 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:07.030421    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.065675 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.065648    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 22 18:14:07.066126 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.065795    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 22 18:14:07.131106 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:07.131079    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.138751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.138712    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 18:09:06 +0000 UTC" deadline="2028-01-19 04:17:47.777566208 +0000 UTC"
Jul 22 18:14:07.138751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.138745    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13090h3m40.638826446s"
Jul 22 18:14:07.170977 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.170951    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 18:14:07.231685 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:07.231622    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.329970 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:07.329934    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb14abac64ed2c3ba96221ac6ba71509b.slice/crio-b23f4caca93cc2540e0b12d746abd3af488e255773afe28d348f3abf0462adaa WatchSource:0}: Error finding container b23f4caca93cc2540e0b12d746abd3af488e255773afe28d348f3abf0462adaa: Status 404 returned error can't find the container with id b23f4caca93cc2540e0b12d746abd3af488e255773afe28d348f3abf0462adaa
Jul 22 18:14:07.332188 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:07.332169    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.335712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.335697    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:14:07.383332 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.383301    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 18:14:07.432730 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:07.432693    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-65.ec2.internal\" not found"
Jul 22 18:14:07.498138 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:07.498112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3fb81ff22ef077bddb6ace0b924312.slice/crio-17cb12225c21c7493d6aeb27eed91f35f3e02caff9f490b581a225e9f7b86f64 WatchSource:0}: Error finding container 17cb12225c21c7493d6aeb27eed91f35f3e02caff9f490b581a225e9f7b86f64: Status 404 returned error can't find the container with id 17cb12225c21c7493d6aeb27eed91f35f3e02caff9f490b581a225e9f7b86f64
Jul 22 18:14:07.509699 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.509677    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 18:14:07.525004 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.524976    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:07.535507 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.535487    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 18:14:07.536557 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.536544    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal"
Jul 22 18:14:07.541997 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:07.541984    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 22 18:14:08.108242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.108207    2580 apiserver.go:52] "Watching apiserver"
Jul 22 18:14:08.115168 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.115147    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 22 18:14:08.115546 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.115522    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-kdvjc","openshift-ovn-kubernetes/ovnkube-node-2rmth","kube-system/global-pull-secret-syncer-2zn89","kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s","openshift-dns/node-resolver-jxvp4","openshift-image-registry/node-ca-mmgns","openshift-multus/multus-9tl6j","openshift-multus/multus-additional-cni-plugins-l2v92","openshift-network-diagnostics/network-check-target-nlzlv","kube-system/konnectivity-agent-s2pqm","openshift-cluster-node-tuning-operator/tuned-rpqwn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal","openshift-multus/network-metrics-daemon-7qrxs"]
Jul 22 18:14:08.117442 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.117424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.119438 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.119420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.119875 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.119853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.120156 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.120141    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rbclh\""
Jul 22 18:14:08.120219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.120159    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.120608 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.120594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:08.120685 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.120661    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:08.121625 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.121606    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 22 18:14:08.121728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.121652    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 22 18:14:08.121778 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.121759    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8wb8c\""
Jul 22 18:14:08.122101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.122083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.123666 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.123647    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.124397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.124380    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.124397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.124391    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 22 18:14:08.124521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.124397    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zkw2v\""
Jul 22 18:14:08.124521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.124390    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.124768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.124755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.125134 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.125114    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 22 18:14:08.125914 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.125864    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.126387 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.125884    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-25p2h\""
Jul 22 18:14:08.126466 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.125930    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.126533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.125938    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 22 18:14:08.127283 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.127239    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 22 18:14:08.127381 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.127307    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.127444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.127391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.127537 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.127519    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.127894 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.127865    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2h4mf\""
Jul 22 18:14:08.128722 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.128704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.129653 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.129635    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 22 18:14:08.129762 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.129750    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.129830 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.129801    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sp49p\""
Jul 22 18:14:08.129962 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.129947    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 22 18:14:08.130040 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.130025    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.130238 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.130223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.130320 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.130302    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:08.130660 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.130643    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 22 18:14:08.130907 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.130878    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 22 18:14:08.130982 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.130927    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kl2g5\""
Jul 22 18:14:08.131497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.131477    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.133509 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.133463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.135032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135011    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 22 18:14:08.135409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135388    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 22 18:14:08.135498 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135423    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135628    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135740    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135948    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.136039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p2r2k\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.136056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.135927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.136217    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwr9n\""
Jul 22 18:14:08.136360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.136313    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 22 18:14:08.137436 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a14c924-510b-4134-a5f1-751bc9baf02d-konnectivity-ca\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.137524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsjs\" (UniqueName: \"kubernetes.io/projected/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-kube-api-access-rvsjs\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.137524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a14c924-510b-4134-a5f1-751bc9baf02d-agent-certs\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.137643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-iptables-alerter-script\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.137643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ae127f4-a41c-4faa-8156-7ec3d2c46382-serviceca\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.137643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137583    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-registration-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.137643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16f3208c-6065-4712-9296-87de0e812da9-hosts-file\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.137643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnld\" (UniqueName: \"kubernetes.io/projected/16f3208c-6065-4712-9296-87de0e812da9-kube-api-access-nsnld\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.137867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137648    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:08.137867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137669    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-device-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.137867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-etc-selinux\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.137867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.137829    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtr2\" (UniqueName: \"kubernetes.io/projected/2a5094cb-5820-4aec-a051-957bb10b6b85-kube-api-access-jgtr2\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.138174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f3208c-6065-4712-9296-87de0e812da9-tmp-dir\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.138174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-host-slash\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.138174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae127f4-a41c-4faa-8156-7ec3d2c46382-host\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.138174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcq8\" (UniqueName: \"kubernetes.io/projected/6ae127f4-a41c-4faa-8156-7ec3d2c46382-kube-api-access-vwcq8\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.138349 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138229    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.138349 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-socket-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.138349 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.138342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-sys-fs\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.140274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.140253    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 22 18:14:08.168690 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.168661    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2kbzj"
Jul 22 18:14:08.174626 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.174564    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2kbzj"
Jul 22 18:14:08.226130 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.226098    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 22 18:14:08.238785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238751    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-socket-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238792    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-tuned\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsjs\" (UniqueName: \"kubernetes.io/projected/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-kube-api-access-rvsjs\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-node-log\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjt8h\" (UniqueName: \"kubernetes.io/projected/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-kube-api-access-wjt8h\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.238945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-k8s-cni-cncf-io\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238953    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-netns\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-socket-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.238979    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-tmp\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-system-cni-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-bin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16f3208c-6065-4712-9296-87de0e812da9-hosts-file\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-etc-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/16f3208c-6065-4712-9296-87de0e812da9-hosts-file\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovn-node-metrics-cert\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysconfig\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.239259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-systemd-units\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-ovn\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-env-overrides\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239358    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-script-lib\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-conf\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239458    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-system-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-run\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-var-lib-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239591    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-bin\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239620    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.239759 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-sys\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-var-lib-kubelet\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a14c924-510b-4134-a5f1-751bc9baf02d-agent-certs\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239814    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ae127f4-a41c-4faa-8156-7ec3d2c46382-serviceca\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-kubelet\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-slash\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-os-release\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-dbus\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.239974    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-etc-selinux\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-os-release\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-netns\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-kubelet\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240108    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-host-slash\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240149    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-conf-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-etc-selinux\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240193    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 22 18:14:08.240578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240201    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-host-slash\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-log-socket\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-kubelet-config\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-multus\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-registration-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240371    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-systemd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240385    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ae127f4-a41c-4faa-8156-7ec3d2c46382-serviceca\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-netd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-config\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-cni-binary-copy\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-registration-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-lib-modules\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzg22\" (UniqueName: \"kubernetes.io/projected/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-kube-api-access-lzg22\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnld\" (UniqueName: \"kubernetes.io/projected/16f3208c-6065-4712-9296-87de0e812da9-kube-api-access-nsnld\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240575    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-device-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.241446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240631    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240660    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-modprobe-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-kubernetes\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-device-dir\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f3208c-6065-4712-9296-87de0e812da9-tmp-dir\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae127f4-a41c-4faa-8156-7ec3d2c46382-host\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240792    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcq8\" (UniqueName: \"kubernetes.io/projected/6ae127f4-a41c-4faa-8156-7ec3d2c46382-kube-api-access-vwcq8\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ae127f4-a41c-4faa-8156-7ec3d2c46382-host\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.240908    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-sys-fs\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16f3208c-6065-4712-9296-87de0e812da9-tmp-dir\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cnibin\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241105    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5094cb-5820-4aec-a051-957bb10b6b85-sys-fs\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241140    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-multus-certs\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-host\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a14c924-510b-4134-a5f1-751bc9baf02d-konnectivity-ca\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.242240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-systemd\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-socket-dir-parent\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-etc-kubernetes\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241297    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vknb\" (UniqueName: \"kubernetes.io/projected/f5c99fd1-d6ff-4257-a696-e3905160793d-kube-api-access-5vknb\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241329    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-iptables-alerter-script\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhtx\" (UniqueName: \"kubernetes.io/projected/3161c822-0d62-468b-9001-8ed2f861e17d-kube-api-access-wdhtx\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-cnibin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtr2\" (UniqueName: \"kubernetes.io/projected/2a5094cb-5820-4aec-a051-957bb10b6b85-kube-api-access-jgtr2\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxsw\" (UniqueName: \"kubernetes.io/projected/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-kube-api-access-qhxsw\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-hostroot\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-daemon-config\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.241906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a14c924-510b-4134-a5f1-751bc9baf02d-konnectivity-ca\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.242944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.242028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-iptables-alerter-script\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.244498 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.244474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a14c924-510b-4134-a5f1-751bc9baf02d-agent-certs\") pod \"konnectivity-agent-s2pqm\" (UID: \"5a14c924-510b-4134-a5f1-751bc9baf02d\") " pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.247602 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.247580    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:08.247735 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.247608    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:08.247735 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.247626    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:08.247735 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.247693    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:08.747673502 +0000 UTC m=+2.963320888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:08.250842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.250794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsjs\" (UniqueName: \"kubernetes.io/projected/73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08-kube-api-access-rvsjs\") pod \"iptables-alerter-kdvjc\" (UID: \"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08\") " pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.251349 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.251323    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnld\" (UniqueName: \"kubernetes.io/projected/16f3208c-6065-4712-9296-87de0e812da9-kube-api-access-nsnld\") pod \"node-resolver-jxvp4\" (UID: \"16f3208c-6065-4712-9296-87de0e812da9\") " pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.251800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.251775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcq8\" (UniqueName: \"kubernetes.io/projected/6ae127f4-a41c-4faa-8156-7ec3d2c46382-kube-api-access-vwcq8\") pod \"node-ca-mmgns\" (UID: \"6ae127f4-a41c-4faa-8156-7ec3d2c46382\") " pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.252325 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.252303    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtr2\" (UniqueName: \"kubernetes.io/projected/2a5094cb-5820-4aec-a051-957bb10b6b85-kube-api-access-jgtr2\") pod \"aws-ebs-csi-driver-node-4s25s\" (UID: \"2a5094cb-5820-4aec-a051-957bb10b6b85\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.320824 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.320763    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal" event={"ID":"0d3fb81ff22ef077bddb6ace0b924312","Type":"ContainerStarted","Data":"17cb12225c21c7493d6aeb27eed91f35f3e02caff9f490b581a225e9f7b86f64"}
Jul 22 18:14:08.322831 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.322799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal" event={"ID":"b14abac64ed2c3ba96221ac6ba71509b","Type":"ContainerStarted","Data":"b23f4caca93cc2540e0b12d746abd3af488e255773afe28d348f3abf0462adaa"}
Jul 22 18:14:08.342306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342245    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-system-cni-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.342306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-bin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-etc-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovn-node-metrics-cert\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342289    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-system-cni-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysconfig\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-systemd-units\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-bin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-ovn\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-env-overrides\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-script-lib\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.342535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-conf\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-system-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-run\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-var-lib-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342643    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-bin\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342782    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-sys\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-var-lib-kubelet\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-kubelet\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-slash\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-os-release\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342958    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-dbus\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.342986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-os-release\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343010    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-netns\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-kubelet\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343047    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-run\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-conf-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343111    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysconfig\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343118    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-log-socket\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343145    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343159    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-systemd-units\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-kubelet-config\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-multus\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-ovn\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-systemd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-netd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343282    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-config\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-cni-binary-copy\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-lib-modules\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.343968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343376    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzg22\" (UniqueName: \"kubernetes.io/projected/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-kube-api-access-lzg22\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-modprobe-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-kubernetes\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cnibin\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343572    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-multus-certs\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-host\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343630    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-systemd\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343652    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-env-overrides\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-socket-dir-parent\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-etc-kubernetes\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vknb\" (UniqueName: \"kubernetes.io/projected/f5c99fd1-d6ff-4257-a696-e3905160793d-kube-api-access-5vknb\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.343725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-socket-dir-parent\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-var-lib-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-bin\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.344817 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.344201    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.344265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:08.844244046 +0000 UTC m=+3.059891432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-binary-copy\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-system-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-etc-kubernetes\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344457    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-etc-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-systemd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.344588    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.344637    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:08.844619571 +0000 UTC m=+3.060266962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-cni-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344831    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-sys\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-var-lib-kubelet\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344916    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-kubelet\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.344990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-cni-netd\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345002    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-slash\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.346829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-os-release\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345124    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-dbus\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-os-release\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345226    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-netns\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345271    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-kubelet\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345471    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-conf-dir\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-config\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345516    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-log-socket\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345551    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cnibin\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345569    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345855    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-ovn-kubernetes\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.345975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-modprobe-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346034    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-kubernetes\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-multus-certs\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346127    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-cni-binary-copy\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346133    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-d\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.348065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhtx\" (UniqueName: \"kubernetes.io/projected/3161c822-0d62-468b-9001-8ed2f861e17d-kube-api-access-wdhtx\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-cnibin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-var-lib-cni-multus\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxsw\" (UniqueName: \"kubernetes.io/projected/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-kube-api-access-qhxsw\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346209    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-systemd\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-hostroot\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346252    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/60a152e6-60d1-4e08-86c5-8d03303c4cec-kubelet-config\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-daemon-config\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovn-node-metrics-cert\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346350    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-tuned\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-sysctl-conf\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-lib-modules\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-host\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-cnibin\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-run-openvswitch\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346703    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-hostroot\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f5c99fd1-d6ff-4257-a696-e3905160793d-multus-daemon-config\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.349068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346839    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-node-log\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjt8h\" (UniqueName: \"kubernetes.io/projected/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-kube-api-access-wjt8h\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346923    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-k8s-cni-cncf-io\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-node-log\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-netns\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.346976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-tmp\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.347184    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f5c99fd1-d6ff-4257-a696-e3905160793d-host-run-k8s-cni-cncf-io\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.347230    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-ovnkube-script-lib\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.350032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.347248    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-host-run-netns\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.351012 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.350991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-etc-tuned\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.352986 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.352940    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-tmp\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.353653 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.353626    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vknb\" (UniqueName: \"kubernetes.io/projected/f5c99fd1-d6ff-4257-a696-e3905160793d-kube-api-access-5vknb\") pod \"multus-9tl6j\" (UID: \"f5c99fd1-d6ff-4257-a696-e3905160793d\") " pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.361555 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.361174    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzg22\" (UniqueName: \"kubernetes.io/projected/156afb27-3c68-40d9-a98c-bb2d3acd9e9b-kube-api-access-lzg22\") pod \"tuned-rpqwn\" (UID: \"156afb27-3c68-40d9-a98c-bb2d3acd9e9b\") " pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.363235 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.362326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhtx\" (UniqueName: \"kubernetes.io/projected/3161c822-0d62-468b-9001-8ed2f861e17d-kube-api-access-wdhtx\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.363367 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.363328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjt8h\" (UniqueName: \"kubernetes.io/projected/523ee8bc-1dd6-49ea-931b-ad3a4ec1791b-kube-api-access-wjt8h\") pod \"ovnkube-node-2rmth\" (UID: \"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b\") " pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.363671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.363649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxsw\" (UniqueName: \"kubernetes.io/projected/b5633a92-24af-4a9f-a530-fcb8d0b9ad21-kube-api-access-qhxsw\") pod \"multus-additional-cni-plugins-l2v92\" (UID: \"b5633a92-24af-4a9f-a530-fcb8d0b9ad21\") " pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.429800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.429761    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jxvp4"
Jul 22 18:14:08.437548 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.437335    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:08.443635 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.443377    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f3208c_6065_4712_9296_87de0e812da9.slice/crio-937902dfb77f4378ae8d69b9c05a79a55ac1e8d458271b3639b342c749817953 WatchSource:0}: Error finding container 937902dfb77f4378ae8d69b9c05a79a55ac1e8d458271b3639b342c749817953: Status 404 returned error can't find the container with id 937902dfb77f4378ae8d69b9c05a79a55ac1e8d458271b3639b342c749817953
Jul 22 18:14:08.446414 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.446388    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kdvjc"
Jul 22 18:14:08.450519 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.450499    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mmgns"
Jul 22 18:14:08.455017 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.454986    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a14c924_510b_4134_a5f1_751bc9baf02d.slice/crio-e4281396e86bdc1624b5b44a945848442ceae1eb16fc062032d65b8a01b777ae WatchSource:0}: Error finding container e4281396e86bdc1624b5b44a945848442ceae1eb16fc062032d65b8a01b777ae: Status 404 returned error can't find the container with id e4281396e86bdc1624b5b44a945848442ceae1eb16fc062032d65b8a01b777ae
Jul 22 18:14:08.456572 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.456375    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s"
Jul 22 18:14:08.460172 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.460150    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73caf5fe_3ffb_43fc_9c2a_8a87f2bc9a08.slice/crio-c94ba953ae698c12101feee19b000f3bdfe4c31f98140900a4ef4d709a311a85 WatchSource:0}: Error finding container c94ba953ae698c12101feee19b000f3bdfe4c31f98140900a4ef4d709a311a85: Status 404 returned error can't find the container with id c94ba953ae698c12101feee19b000f3bdfe4c31f98140900a4ef4d709a311a85
Jul 22 18:14:08.463133 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.463008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9tl6j"
Jul 22 18:14:08.467419 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.467395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae127f4_a41c_4faa_8156_7ec3d2c46382.slice/crio-0191371957b5e1393ed15fa09d98d8c66640286e6719146467f6bad6f12a2cd9 WatchSource:0}: Error finding container 0191371957b5e1393ed15fa09d98d8c66640286e6719146467f6bad6f12a2cd9: Status 404 returned error can't find the container with id 0191371957b5e1393ed15fa09d98d8c66640286e6719146467f6bad6f12a2cd9
Jul 22 18:14:08.469554 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.469533    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5094cb_5820_4aec_a051_957bb10b6b85.slice/crio-658f635cf9b57c40d5ca1d0613baeed622382195b898b8cb41809c9046d69ea3 WatchSource:0}: Error finding container 658f635cf9b57c40d5ca1d0613baeed622382195b898b8cb41809c9046d69ea3: Status 404 returned error can't find the container with id 658f635cf9b57c40d5ca1d0613baeed622382195b898b8cb41809c9046d69ea3
Jul 22 18:14:08.471633 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.471591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l2v92"
Jul 22 18:14:08.477100 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.477065    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c99fd1_d6ff_4257_a696_e3905160793d.slice/crio-808b87c3fdb7f0de7e32f0581d3f3563417f0c36e3d395d21fe745a13e936d74 WatchSource:0}: Error finding container 808b87c3fdb7f0de7e32f0581d3f3563417f0c36e3d395d21fe745a13e936d74: Status 404 returned error can't find the container with id 808b87c3fdb7f0de7e32f0581d3f3563417f0c36e3d395d21fe745a13e936d74
Jul 22 18:14:08.478201 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.477585    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:08.482788 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.482764    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5633a92_24af_4a9f_a530_fcb8d0b9ad21.slice/crio-e69028751a4a1e9669a0e48575d5c3fd5b63cf854682814c9240b995ed9e73df WatchSource:0}: Error finding container e69028751a4a1e9669a0e48575d5c3fd5b63cf854682814c9240b995ed9e73df: Status 404 returned error can't find the container with id e69028751a4a1e9669a0e48575d5c3fd5b63cf854682814c9240b995ed9e73df
Jul 22 18:14:08.483611 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.483589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rpqwn"
Jul 22 18:14:08.487959 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.487934    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523ee8bc_1dd6_49ea_931b_ad3a4ec1791b.slice/crio-402230ac02b8aaa93688f1e957468cc6764270a5537cf8e64a47679d2a294762 WatchSource:0}: Error finding container 402230ac02b8aaa93688f1e957468cc6764270a5537cf8e64a47679d2a294762: Status 404 returned error can't find the container with id 402230ac02b8aaa93688f1e957468cc6764270a5537cf8e64a47679d2a294762
Jul 22 18:14:08.493995 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:08.493966    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156afb27_3c68_40d9_a98c_bb2d3acd9e9b.slice/crio-6d9fa2c7675bf024c52e77b8f93a037bb745e9235fddd84cb9aecf4660b5a761 WatchSource:0}: Error finding container 6d9fa2c7675bf024c52e77b8f93a037bb745e9235fddd84cb9aecf4660b5a761: Status 404 returned error can't find the container with id 6d9fa2c7675bf024c52e77b8f93a037bb745e9235fddd84cb9aecf4660b5a761
Jul 22 18:14:08.533038 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.533007    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 22 18:14:08.750288 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.750219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:08.750510 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.750436    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:08.750510 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.750464    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:08.750510 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.750479    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:08.750693 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.750540    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:09.750521615 +0000 UTC m=+3.966168999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:08.851399 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.851355    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:08.851399 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:08.851406    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:08.851630 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.851546    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:08.851630 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.851607    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:09.851593233 +0000 UTC m=+4.067240615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:08.851744 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.851547    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:08.851744 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:08.851727    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:09.851709861 +0000 UTC m=+4.067357262 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:09.176858 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.176740    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:09:08 +0000 UTC" deadline="2028-02-24 09:17:44.618977975 +0000 UTC"
Jul 22 18:14:09.176858 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.176782    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13959h3m35.442200965s"
Jul 22 18:14:09.315480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.314924    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:09.315480 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.315087    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:09.315480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.315389    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:09.315793 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.315503    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:09.328256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.328219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerStarted","Data":"e69028751a4a1e9669a0e48575d5c3fd5b63cf854682814c9240b995ed9e73df"}
Jul 22 18:14:09.330858 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.330779    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9tl6j" event={"ID":"f5c99fd1-d6ff-4257-a696-e3905160793d","Type":"ContainerStarted","Data":"808b87c3fdb7f0de7e32f0581d3f3563417f0c36e3d395d21fe745a13e936d74"}
Jul 22 18:14:09.332386 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.332312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmgns" event={"ID":"6ae127f4-a41c-4faa-8156-7ec3d2c46382","Type":"ContainerStarted","Data":"0191371957b5e1393ed15fa09d98d8c66640286e6719146467f6bad6f12a2cd9"}
Jul 22 18:14:09.342404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.342362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdvjc" event={"ID":"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08","Type":"ContainerStarted","Data":"c94ba953ae698c12101feee19b000f3bdfe4c31f98140900a4ef4d709a311a85"}
Jul 22 18:14:09.344992 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.344961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2pqm" event={"ID":"5a14c924-510b-4134-a5f1-751bc9baf02d","Type":"ContainerStarted","Data":"e4281396e86bdc1624b5b44a945848442ceae1eb16fc062032d65b8a01b777ae"}
Jul 22 18:14:09.347092 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.347065    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpqwn" event={"ID":"156afb27-3c68-40d9-a98c-bb2d3acd9e9b","Type":"ContainerStarted","Data":"6d9fa2c7675bf024c52e77b8f93a037bb745e9235fddd84cb9aecf4660b5a761"}
Jul 22 18:14:09.349816 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.349791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s" event={"ID":"2a5094cb-5820-4aec-a051-957bb10b6b85","Type":"ContainerStarted","Data":"658f635cf9b57c40d5ca1d0613baeed622382195b898b8cb41809c9046d69ea3"}
Jul 22 18:14:09.352027 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.352002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jxvp4" event={"ID":"16f3208c-6065-4712-9296-87de0e812da9","Type":"ContainerStarted","Data":"937902dfb77f4378ae8d69b9c05a79a55ac1e8d458271b3639b342c749817953"}
Jul 22 18:14:09.353639 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.353541    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"402230ac02b8aaa93688f1e957468cc6764270a5537cf8e64a47679d2a294762"}
Jul 22 18:14:09.759121 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.758226    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:09.759121 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.758452    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:09.759121 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.758503    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:09.759121 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.758519    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:09.759121 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.758589    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:11.758569799 +0000 UTC m=+5.974217185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:09.859446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.859408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:09.859624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:09.859463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:09.859624 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.859597    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:09.859770 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.859667    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:11.859644692 +0000 UTC m=+6.075292078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:09.860228 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.860095    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:09.860228 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:09.860154    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:11.860136484 +0000 UTC m=+6.075783898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:10.177528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:10.177430    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 18:09:08 +0000 UTC" deadline="2028-01-07 16:44:28.132862665 +0000 UTC"
Jul 22 18:14:10.177528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:10.177495    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12814h30m17.955371853s"
Jul 22 18:14:10.315277 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:10.314724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:10.315277 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:10.314883    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:11.314696 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:11.314659    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:11.315227 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.314807    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:11.315297 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:11.315253    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:11.315418 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.315368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:11.780443 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:11.780399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:11.780628 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.780574    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:11.780628 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.780598    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:11.780628 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.780612    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:11.780819 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.780675    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:15.780654532 +0000 UTC m=+9.996301932 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:11.881177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:11.881141    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:11.881371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:11.881192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:11.881371 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.881319    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:11.881486 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.881380    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:15.881361162 +0000 UTC m=+10.097008551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:11.881810 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.881790    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:11.881914 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:11.881852    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:15.881834236 +0000 UTC m=+10.097481626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:12.315533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:12.315497    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:12.316054 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:12.315646    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:13.314790 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:13.314753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:13.314982 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:13.314753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:13.314982 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:13.314915    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:13.314982 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:13.314967    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:14.315416 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:14.315380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:14.316118 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:14.315530    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:15.314692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:15.314656    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:15.314944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:15.314698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:15.314944 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.314823    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:15.315061 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.314949    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:15.818620 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:15.818578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:15.819152 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.818783    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:15.819152 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.818821    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:15.819152 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.818836    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:15.819152 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.818897    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:23.818878012 +0000 UTC m=+18.034525407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:15.919256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:15.919198    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:15.919256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:15.919250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:15.919498 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.919414    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:15.919498 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.919479    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:23.919460041 +0000 UTC m=+18.135107423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:15.919832 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.919716    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:15.919832 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:15.919774    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:23.919758499 +0000 UTC m=+18.135405888 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:16.315641 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:16.315604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:16.315839 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:16.315723    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:17.314838 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:17.314790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:17.315276 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:17.314839    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:17.315276 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:17.314945    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:17.315446 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:17.315406    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:18.315143 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:18.315106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:18.315541 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:18.315257    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:19.315015 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:19.314984    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:19.315201 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:19.315028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:19.315201 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:19.315115    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:19.315570 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:19.315250    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:20.314354 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:20.314314    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:20.314539 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:20.314462    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:21.314444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:21.314410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:21.314896 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:21.314450    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:21.314896 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:21.314531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:21.314896 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:21.314668    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:22.314663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:22.314628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:22.315121 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:22.314759    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:23.314350 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:23.314318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:23.314525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:23.314318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:23.314525 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.314410    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:23.314525 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.314499    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:23.875648 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:23.875610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:23.876096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.875769    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:23.876096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.875792    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:23.876096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.875817    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:23.876096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.875880    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:14:39.875863293 +0000 UTC m=+34.091510674 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:23.976775 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:23.976742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:23.976775 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:23.976778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:23.976977 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.976926    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:23.977028 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.976989    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:14:39.976972745 +0000 UTC m=+34.192620126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:23.977028 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.976924    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:23.977116 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:23.977076    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:39.977058089 +0000 UTC m=+34.192705470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:24.314664 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:24.314629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:24.314871 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:24.314758    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:25.314372 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:25.314337    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:25.314773 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:25.314392    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:25.314773 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:25.314484    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:25.314773 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:25.314573    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:26.315822 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:26.315774    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:26.316305 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:26.315898    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:27.314793 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:27.314753    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:27.314969 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:27.314765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:27.314969 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:27.314906    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:27.315082 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:27.315057    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:28.315231 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.314939    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:28.315786 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:28.315305    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:28.394049 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.394020    2580 generic.go:356] "Generic (PLEG): container finished" podID="0d3fb81ff22ef077bddb6ace0b924312" containerID="dd084dd82ce80288374ed8c1831c787f2bc7fb5928985d8734e67dd8aaf6aec4" exitCode=0
Jul 22 18:14:28.394169 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.394097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal" event={"ID":"0d3fb81ff22ef077bddb6ace0b924312","Type":"ContainerDied","Data":"dd084dd82ce80288374ed8c1831c787f2bc7fb5928985d8734e67dd8aaf6aec4"}
Jul 22 18:14:28.395505 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.395483    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal" event={"ID":"b14abac64ed2c3ba96221ac6ba71509b","Type":"ContainerStarted","Data":"95c1fdec56e44bebd3531f1b0c19a2e7ed8a424f82aa68f8c2ec435232e9e591"}
Jul 22 18:14:28.398470 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"f372439aa35f1b61e03b9d968dbc58caacd47f7707045475c275739bde8fb69f"}
Jul 22 18:14:28.398548 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"03029e7fafbfc8785abf997450a6d0de55f06b2e6c6e24a4fca99c94c4137700"}
Jul 22 18:14:28.398592 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"599400e90c9179989da159a510af7e2f0e4eb862d3e42a1a45bb5a2d6b7aaa6c"}
Jul 22 18:14:28.398637 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"c143c1df0a69375cf466bc5f539944b4e60fd9b8ab8dfbea11130a0a6b78ecde"}
Jul 22 18:14:28.398637 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"93934de63d99a67b20ef8b2b293ed80b7779322a189579b9fb39277f63a311c2"}
Jul 22 18:14:28.398704 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.398664    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"4d7f0a3736a177f6702e50227d8015d78130671bbcf23b81a4260050f9aecf7b"}
Jul 22 18:14:28.399717 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.399699    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="efb2495fb90d6c3e675280959a3db2e6cce31696ad2af786767508d29d58eba0" exitCode=0
Jul 22 18:14:28.399784 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.399758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"efb2495fb90d6c3e675280959a3db2e6cce31696ad2af786767508d29d58eba0"}
Jul 22 18:14:28.401162 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.401143    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9tl6j" event={"ID":"f5c99fd1-d6ff-4257-a696-e3905160793d","Type":"ContainerStarted","Data":"774df1649aac6f0eee44ba85bff8294d04db41fcc170440b8936965ff0a7f542"}
Jul 22 18:14:28.402564 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.402543    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mmgns" event={"ID":"6ae127f4-a41c-4faa-8156-7ec3d2c46382","Type":"ContainerStarted","Data":"85212400239e670b4482ae1d88e946fef27534c497a1a97314a178952e593d4b"}
Jul 22 18:14:28.403948 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.403928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s2pqm" event={"ID":"5a14c924-510b-4134-a5f1-751bc9baf02d","Type":"ContainerStarted","Data":"6ccbae2c0daa142615b3458c6c3b66e4161a20bb63fa969f8b4e9abd859b7218"}
Jul 22 18:14:28.405271 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.405245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rpqwn" event={"ID":"156afb27-3c68-40d9-a98c-bb2d3acd9e9b","Type":"ContainerStarted","Data":"7e17f56508a6c0c06e4282569028c2416c264fde455d8e87931706f2f0ef6c43"}
Jul 22 18:14:28.406570 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.406553    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s" event={"ID":"2a5094cb-5820-4aec-a051-957bb10b6b85","Type":"ContainerStarted","Data":"2887d7c035ce91e0bff2206c36f131db66c7014b0603f9dbec17d8655820a5a3"}
Jul 22 18:14:28.407722 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.407704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jxvp4" event={"ID":"16f3208c-6065-4712-9296-87de0e812da9","Type":"ContainerStarted","Data":"125ce573b328a7aeadeb9760b8c88d882dda031100bb0b0c7cdfaad615282ba8"}
Jul 22 18:14:28.421026 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.420976    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mmgns" podStartSLOduration=11.04278054 podStartE2EDuration="21.420961939s" totalImagesPullingTime="10.378181399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.420716336 +0000 UTC m=+22.636363739" watchObservedRunningTime="2026-07-22 18:14:28.420961939 +0000 UTC m=+22.636609356"
Jul 22 18:14:28.455268 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.455223    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-65.ec2.internal" podStartSLOduration=20.455205618 podStartE2EDuration="20.455205618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.454868081 +0000 UTC m=+22.670515485" watchObservedRunningTime="2026-07-22 18:14:28.455205618 +0000 UTC m=+22.670853021"
Jul 22 18:14:28.455564 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.455524    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9tl6j" podStartSLOduration=2.382456214 podStartE2EDuration="21.455514822s" totalImagesPullingTime="19.073058608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.43632493 +0000 UTC m=+22.651972335" watchObservedRunningTime="2026-07-22 18:14:28.455514822 +0000 UTC m=+22.671162225"
Jul 22 18:14:28.495317 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.495069    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rpqwn" podStartSLOduration=2.441956772 podStartE2EDuration="21.495056084s" totalImagesPullingTime="19.053099312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.494721307 +0000 UTC m=+22.710368711" watchObservedRunningTime="2026-07-22 18:14:28.495056084 +0000 UTC m=+22.710703486"
Jul 22 18:14:28.506624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.506591    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:28.507103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.507085    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:28.507783 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.507747    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jxvp4" podStartSLOduration=7.802566594 podStartE2EDuration="22.507734864s" totalImagesPullingTime="14.70516827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.507622102 +0000 UTC m=+22.723269505" watchObservedRunningTime="2026-07-22 18:14:28.507734864 +0000 UTC m=+22.723382266"
Jul 22 18:14:28.521701 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:28.521668    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s2pqm" podStartSLOduration=3.473349094 podStartE2EDuration="22.521656172s" totalImagesPullingTime="19.048307078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:28.521445666 +0000 UTC m=+22.737093069" watchObservedRunningTime="2026-07-22 18:14:28.521656172 +0000 UTC m=+22.737303576"
Jul 22 18:14:29.315164 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.315132    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:29.315450 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:29.315275    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:29.315450 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.315322    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:29.315450 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:29.315437    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:29.412193 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.412161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal" event={"ID":"0d3fb81ff22ef077bddb6ace0b924312","Type":"ContainerStarted","Data":"5c0e520acbb3c4119cefe948df2a9295785c2685bf3c3b64de5f44136443b446"}
Jul 22 18:14:29.413865 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.413837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kdvjc" event={"ID":"73caf5fe-3ffb-43fc-9c2a-8a87f2bc9a08","Type":"ContainerStarted","Data":"7b502cf5164bb55c2d8c33d6ba2a3ffcf7a32c704241513ca1a46b15225b54ab"}
Jul 22 18:14:29.427542 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.427488    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-65.ec2.internal" podStartSLOduration=21.427469984 podStartE2EDuration="21.427469984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:29.426846456 +0000 UTC m=+23.642493860" watchObservedRunningTime="2026-07-22 18:14:29.427469984 +0000 UTC m=+23.643117387"
Jul 22 18:14:29.441326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.441281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kdvjc" podStartSLOduration=4.398203835 podStartE2EDuration="23.441263312s" totalImagesPullingTime="19.043059477s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:29.440770151 +0000 UTC m=+23.656417585" watchObservedRunningTime="2026-07-22 18:14:29.441263312 +0000 UTC m=+23.656910717"
Jul 22 18:14:29.609940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:29.609915    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 22 18:14:30.202838 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.202675    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T18:14:29.609937208Z","UUID":"8aeb4de1-1fbb-4602-9319-647987e062f7","Handler":null,"Name":"","Endpoint":""}
Jul 22 18:14:30.204555 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.204531    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 22 18:14:30.204699 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.204584    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 22 18:14:30.314281 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.314252    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:30.314459 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:30.314385    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:30.419222 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.419184    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s" event={"ID":"2a5094cb-5820-4aec-a051-957bb10b6b85","Type":"ContainerStarted","Data":"a3f4034727fd21ec5fc1c00ab728f994b9a3cc94f646306e1ba3c340a99d4329"}
Jul 22 18:14:30.419222 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:30.419209    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 22 18:14:31.314899 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:31.314862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:31.315105 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:31.314862    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:31.315105 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:31.315013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:31.315105 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:31.315056    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:31.425387 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:31.425348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"ff720874acb8dd2c91186502ceaa7a43e77da091dcb355add1db6781943b2dc6"}
Jul 22 18:14:32.314953 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:32.314923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:32.315123 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:32.315049    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:33.314843 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.314652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:33.315583 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.314663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:33.315583 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:33.314939    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:33.315583 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:33.315044    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:33.433787 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.433756    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" event={"ID":"523ee8bc-1dd6-49ea-931b-ad3a4ec1791b","Type":"ContainerStarted","Data":"cfbbbbb75ce5abc1b72e9143c63bcc4f5a221b3b79b436f24ea4e151c57f530a"}
Jul 22 18:14:33.434112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.434086    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:33.434222 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.434164    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:33.434222 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.434210    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:33.435961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.435934    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="23645ce5ae7c1edc84af32bc0df39ca74b38b82e2392447247ab047dea192552" exitCode=0
Jul 22 18:14:33.436059 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.436009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"23645ce5ae7c1edc84af32bc0df39ca74b38b82e2392447247ab047dea192552"}
Jul 22 18:14:33.438655 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.438633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s" event={"ID":"2a5094cb-5820-4aec-a051-957bb10b6b85","Type":"ContainerStarted","Data":"2bbedbf848955c89397eb87f2f8833e3c5f00098c3200083b6f71084000836e2"}
Jul 22 18:14:33.451458 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.450209    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:33.451458 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.450398    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:14:33.457983 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.457945    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth" podStartSLOduration=7.292074028 podStartE2EDuration="26.457934206s" totalImagesPullingTime="19.165860178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:33.457783243 +0000 UTC m=+27.673430647" watchObservedRunningTime="2026-07-22 18:14:33.457934206 +0000 UTC m=+27.673581609"
Jul 22 18:14:33.473738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:33.473693    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4s25s" podStartSLOduration=7.254181717 podStartE2EDuration="26.473683065s" totalImagesPullingTime="19.219501348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:33.473619234 +0000 UTC m=+27.689266637" watchObservedRunningTime="2026-07-22 18:14:33.473683065 +0000 UTC m=+27.689330465"
Jul 22 18:14:34.315071 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:34.315036    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:34.315531 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:34.315177    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:35.008559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.008352    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:35.008738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.008656    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 22 18:14:35.008973 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.008955    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s2pqm"
Jul 22 18:14:35.120867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.120836    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qrxs"]
Jul 22 18:14:35.120995 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.120954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:35.121053 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:35.121036    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:35.123308 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.123265    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2zn89"]
Jul 22 18:14:35.123308 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.123372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:35.123580 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:35.123460    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:35.123993 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.123964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlzlv"]
Jul 22 18:14:35.124071 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.124037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:35.124118 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:35.124107    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:35.444855 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.444818    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="86a17d9f009ee78c96e1823afa8f6687ab4f312a1e38d8bf2893f4fedf656065" exitCode=0
Jul 22 18:14:35.444855 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:35.444843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"86a17d9f009ee78c96e1823afa8f6687ab4f312a1e38d8bf2893f4fedf656065"}
Jul 22 18:14:36.315604 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:36.315573    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:36.315768 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:36.315684    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:37.314664 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:37.314637    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:37.314664 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:37.314661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:37.315070 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:37.314742    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:37.315070 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:37.314872    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:37.451028 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:37.450995    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="2fd726b067b8c5fde98b763ccb2d3446fd155b169f84f8304366f5f5fcbe7b3c" exitCode=0
Jul 22 18:14:37.451155 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:37.451054    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"2fd726b067b8c5fde98b763ccb2d3446fd155b169f84f8304366f5f5fcbe7b3c"}
Jul 22 18:14:38.315599 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:38.315263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:38.316062 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:38.315831    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:39.315204 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:39.315174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:39.315380 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:39.315174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:39.315380 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.315280    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzlv" podUID="db97839c-0b2d-44c4-8fa1-aaaed691f651"
Jul 22 18:14:39.315472 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.315376    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:14:39.902565 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:39.902528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:39.903009 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.902706    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 22 18:14:39.903009 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.902726    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 22 18:14:39.903009 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.902736    2580 projected.go:196] Error preparing data for projected volume kube-api-access-mhvk4 for pod openshift-network-diagnostics/network-check-target-nlzlv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:39.903009 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:39.902790    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4 podName:db97839c-0b2d-44c4-8fa1-aaaed691f651 nodeName:}" failed. No retries permitted until 2026-07-22 18:15:11.902775265 +0000 UTC m=+66.118422668 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mhvk4" (UniqueName: "kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4") pod "network-check-target-nlzlv" (UID: "db97839c-0b2d-44c4-8fa1-aaaed691f651") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 22 18:14:40.003417 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.003380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:40.003417 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.003421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:40.003659 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.003564    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:40.003659 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.003626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret podName:60a152e6-60d1-4e08-86c5-8d03303c4cec nodeName:}" failed. No retries permitted until 2026-07-22 18:15:12.003608804 +0000 UTC m=+66.219256201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret") pod "global-pull-secret-syncer-2zn89" (UID: "60a152e6-60d1-4e08-86c5-8d03303c4cec") : object "kube-system"/"original-pull-secret" not registered
Jul 22 18:14:40.003659 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.003564    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:40.003839 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.003717    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:15:12.003699188 +0000 UTC m=+66.219346588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 22 18:14:40.314401 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.314367    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:40.314586 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.314500    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2zn89" podUID="60a152e6-60d1-4e08-86c5-8d03303c4cec"
Jul 22 18:14:40.668669 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.668586    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-65.ec2.internal" event="NodeReady"
Jul 22 18:14:40.668836 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.668735    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 22 18:14:40.718629 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.718448    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bthdp"]
Jul 22 18:14:40.721148 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.721129    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.724528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.724500    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 22 18:14:40.724528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.724512    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ttgch\""
Jul 22 18:14:40.724692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.724542    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 22 18:14:40.732089 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.732068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hgxlm"]
Jul 22 18:14:40.733958 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.733938    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.737086 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.737046    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 22 18:14:40.737177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.737132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7ttjh\""
Jul 22 18:14:40.737233 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.737141    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 22 18:14:40.737545 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.737529    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 22 18:14:40.742211 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.742190    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bthdp"]
Jul 22 18:14:40.752614 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.752590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgxlm"]
Jul 22 18:14:40.807784 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807754    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f06502df-814d-476e-9f23-9eaae5e32a1d-config-volume\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.807968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnp5q\" (UniqueName: \"kubernetes.io/projected/f06502df-814d-476e-9f23-9eaae5e32a1d-kube-api-access-qnp5q\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.807968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807835    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107e48ae-5663-4d7d-9d6d-a26d5270144c-cert\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.807968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807901    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06502df-814d-476e-9f23-9eaae5e32a1d-tmp-dir\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.807968 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807937    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6vqd\" (UniqueName: \"kubernetes.io/projected/107e48ae-5663-4d7d-9d6d-a26d5270144c-kube-api-access-n6vqd\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.808137 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.807969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.909168 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909136    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06502df-814d-476e-9f23-9eaae5e32a1d-tmp-dir\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909177    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6vqd\" (UniqueName: \"kubernetes.io/projected/107e48ae-5663-4d7d-9d6d-a26d5270144c-kube-api-access-n6vqd\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f06502df-814d-476e-9f23-9eaae5e32a1d-config-volume\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.909462    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnp5q\" (UniqueName: \"kubernetes.io/projected/f06502df-814d-476e-9f23-9eaae5e32a1d-kube-api-access-qnp5q\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:40.909524    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:41.40950563 +0000 UTC m=+35.625153020 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:40.909627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107e48ae-5663-4d7d-9d6d-a26d5270144c-cert\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.909996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.909839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f06502df-814d-476e-9f23-9eaae5e32a1d-tmp-dir\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.910188 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.910163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f06502df-814d-476e-9f23-9eaae5e32a1d-config-volume\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.914421 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.914398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/107e48ae-5663-4d7d-9d6d-a26d5270144c-cert\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:40.918019 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.917991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnp5q\" (UniqueName: \"kubernetes.io/projected/f06502df-814d-476e-9f23-9eaae5e32a1d-kube-api-access-qnp5q\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:40.918285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:40.918242    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6vqd\" (UniqueName: \"kubernetes.io/projected/107e48ae-5663-4d7d-9d6d-a26d5270144c-kube-api-access-n6vqd\") pod \"ingress-canary-hgxlm\" (UID: \"107e48ae-5663-4d7d-9d6d-a26d5270144c\") " pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:41.043892 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.043861    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgxlm"
Jul 22 18:14:41.314786 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.314699    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:14:41.315008 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.314830    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:14:41.317530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.317434    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjc8f\""
Jul 22 18:14:41.317530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.317488    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 18:14:41.317530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.317493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 18:14:41.317530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.317493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 18:14:41.317835 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.317602    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8sjds\""
Jul 22 18:14:41.414779 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.414743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:41.414952 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:41.414891    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:41.415015 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:41.414958    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:42.414938324 +0000 UTC m=+36.630585718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:41.559092 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:41.559054    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgxlm"]
Jul 22 18:14:41.563367 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:41.563340    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107e48ae_5663_4d7d_9d6d_a26d5270144c.slice/crio-03eee96bf49a9525aac65001805e70d7fba3dc4b3d38ad4a73b845f76b8cc11f WatchSource:0}: Error finding container 03eee96bf49a9525aac65001805e70d7fba3dc4b3d38ad4a73b845f76b8cc11f: Status 404 returned error can't find the container with id 03eee96bf49a9525aac65001805e70d7fba3dc4b3d38ad4a73b845f76b8cc11f
Jul 22 18:14:42.315298 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.315267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:14:42.318000 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.317963    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 18:14:42.422367 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.422333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:42.422530 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:42.422507    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:42.422605 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:42.422593    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:44.422570234 +0000 UTC m=+38.638217636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:42.465603 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.465569    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="a360141e5028b86952ebf6090c7e89b722c7455d92da04d32e4c91c428b92dab" exitCode=0
Jul 22 18:14:42.465789 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.465675    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"a360141e5028b86952ebf6090c7e89b722c7455d92da04d32e4c91c428b92dab"}
Jul 22 18:14:42.467177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:42.467147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgxlm" event={"ID":"107e48ae-5663-4d7d-9d6d-a26d5270144c","Type":"ContainerStarted","Data":"03eee96bf49a9525aac65001805e70d7fba3dc4b3d38ad4a73b845f76b8cc11f"}
Jul 22 18:14:43.473356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:43.473184    2580 generic.go:356] "Generic (PLEG): container finished" podID="b5633a92-24af-4a9f-a530-fcb8d0b9ad21" containerID="7911d874f8fe0c51fcc4585406869994b2a663fd2ad5dcfbaa508df16ea3c1e5" exitCode=0
Jul 22 18:14:43.473356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:43.473271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerDied","Data":"7911d874f8fe0c51fcc4585406869994b2a663fd2ad5dcfbaa508df16ea3c1e5"}
Jul 22 18:14:44.440285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:44.440241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:44.440428 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:44.440390    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:44.440488 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:44.440465    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:48.440449762 +0000 UTC m=+42.656097142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:44.479230 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:44.479196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l2v92" event={"ID":"b5633a92-24af-4a9f-a530-fcb8d0b9ad21","Type":"ContainerStarted","Data":"2e0dffd39806f7b99997dd1fcf01f37098f7c34ebf116dd0242b8acff551b713"}
Jul 22 18:14:44.480680 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:44.480644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgxlm" event={"ID":"107e48ae-5663-4d7d-9d6d-a26d5270144c","Type":"ContainerStarted","Data":"6323977ff3c973e245722bb5130624edbc6dc954dfab719d1a6b0a63ac9c3243"}
Jul 22 18:14:44.499663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:44.499571    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l2v92" podStartSLOduration=7.693553483 podStartE2EDuration="37.499557664s" totalImagesPullingTime="29.806004181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:44.499242507 +0000 UTC m=+38.714889910" watchObservedRunningTime="2026-07-22 18:14:44.499557664 +0000 UTC m=+38.715205045"
Jul 22 18:14:44.515798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:44.515748    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hgxlm" podStartSLOduration=1.759567989 podStartE2EDuration="4.515731292s" totalImagesPullingTime="2.756163303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:14:44.515629136 +0000 UTC m=+38.731276542" watchObservedRunningTime="2026-07-22 18:14:44.515731292 +0000 UTC m=+38.731378690"
Jul 22 18:14:48.469509 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:48.469440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:48.469912 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:48.469601    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:48.469912 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:48.469664    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:14:56.46964884 +0000 UTC m=+50.685296221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:56.522077 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:56.522044    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:14:56.522563 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:56.522166    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:14:56.522563 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:14:56.522221    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:15:12.5222088 +0000 UTC m=+66.737856180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:14:58.018398 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.018368    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"]
Jul 22 18:14:58.036604 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.036576    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"]
Jul 22 18:14:58.036771 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.036703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.039080 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.039021    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 22 18:14:58.039299 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.039249    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-sbd5z\""
Jul 22 18:14:58.039416 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.039338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 22 18:14:58.039416 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.039406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 22 18:14:58.039970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.039954    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 22 18:14:58.050673 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.050651    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"]
Jul 22 18:14:58.050757 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.050682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"]
Jul 22 18:14:58.050796 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.050765    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.052606 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.052591    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 22 18:14:58.052910 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.052894    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 22 18:14:58.052992 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.052980    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 22 18:14:58.053059 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.053025    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 22 18:14:58.135415 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135370    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf8w\" (UniqueName: \"kubernetes.io/projected/146b460e-963a-46c6-9325-3f35674e287d-kube-api-access-plf8w\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.135587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85524e-af3f-4495-8caf-7f56ed552f68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.135587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.135587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/146b460e-963a-46c6-9325-3f35674e287d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.135587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135584    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.135724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.135724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xls6m\" (UniqueName: \"kubernetes.io/projected/af85524e-af3f-4495-8caf-7f56ed552f68-kube-api-access-xls6m\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.135724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.135672    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236083 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85524e-af3f-4495-8caf-7f56ed552f68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236083 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236125    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/146b460e-963a-46c6-9325-3f35674e287d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236155    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xls6m\" (UniqueName: \"kubernetes.io/projected/af85524e-af3f-4495-8caf-7f56ed552f68-kube-api-access-xls6m\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236191    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.236305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236243    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf8w\" (UniqueName: \"kubernetes.io/projected/146b460e-963a-46c6-9325-3f35674e287d-kube-api-access-plf8w\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.236899 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.236871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/af85524e-af3f-4495-8caf-7f56ed552f68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.240057 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.240028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-ca\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.240163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.240064    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.240163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.240071    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.240163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.240076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af85524e-af3f-4495-8caf-7f56ed552f68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.240307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.240291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/146b460e-963a-46c6-9325-3f35674e287d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.244506 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.244476    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf8w\" (UniqueName: \"kubernetes.io/projected/146b460e-963a-46c6-9325-3f35674e287d-kube-api-access-plf8w\") pod \"managed-serviceaccount-addon-agent-58496bdc46-cgf9t\" (UID: \"146b460e-963a-46c6-9325-3f35674e287d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.244650 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.244636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xls6m\" (UniqueName: \"kubernetes.io/projected/af85524e-af3f-4495-8caf-7f56ed552f68-kube-api-access-xls6m\") pod \"cluster-proxy-proxy-agent-78fb5558fc-tf5hn\" (UID: \"af85524e-af3f-4495-8caf-7f56ed552f68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.358510 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.358431    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:14:58.364620 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.364588    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:14:58.511866 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.511830    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"]
Jul 22 18:14:58.516121 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:58.516090    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod146b460e_963a_46c6_9325_3f35674e287d.slice/crio-e4392fd1326e58baf7e19d79304d633ca844b0010245cd102374326fbdf8197c WatchSource:0}: Error finding container e4392fd1326e58baf7e19d79304d633ca844b0010245cd102374326fbdf8197c: Status 404 returned error can't find the container with id e4392fd1326e58baf7e19d79304d633ca844b0010245cd102374326fbdf8197c
Jul 22 18:14:58.539928 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:58.539904    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"]
Jul 22 18:14:58.553233 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:14:58.553203    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf85524e_af3f_4495_8caf_7f56ed552f68.slice/crio-aca506a9f4b386538002a25de39a8deae78d8a794fdd216b361eee5db8380fba WatchSource:0}: Error finding container aca506a9f4b386538002a25de39a8deae78d8a794fdd216b361eee5db8380fba: Status 404 returned error can't find the container with id aca506a9f4b386538002a25de39a8deae78d8a794fdd216b361eee5db8380fba
Jul 22 18:14:59.520544 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:59.520502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerStarted","Data":"aca506a9f4b386538002a25de39a8deae78d8a794fdd216b361eee5db8380fba"}
Jul 22 18:14:59.521716 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:14:59.521689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t" event={"ID":"146b460e-963a-46c6-9325-3f35674e287d","Type":"ContainerStarted","Data":"e4392fd1326e58baf7e19d79304d633ca844b0010245cd102374326fbdf8197c"}
Jul 22 18:15:05.457727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:05.457696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2rmth"
Jul 22 18:15:07.545901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.545797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerStarted","Data":"6abae5d1233a16dfed8bde1eae4c87e52460d4b9d15a9a857a41966282d94fd5"}
Jul 22 18:15:07.545901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.545863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerStarted","Data":"87669a9db1e958594f2f80a6e4a3ba4b021553b6ea09a5ff4fda297efe54a5a1"}
Jul 22 18:15:07.545901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.545880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerStarted","Data":"118bf7f9e2fd07cf91585a9080c6e5b20b00085c537af44aed20eb7006a0c121"}
Jul 22 18:15:07.547137 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.547114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t" event={"ID":"146b460e-963a-46c6-9325-3f35674e287d","Type":"ContainerStarted","Data":"1849d6f38b1a32eeed76e9f9970f204dee1068de3e15d4eec81684fc22221992"}
Jul 22 18:15:07.563057 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.562996    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" podStartSLOduration=0.866272774 podStartE2EDuration="9.562983194s" totalImagesPullingTime="8.69671042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:15:07.56210872 +0000 UTC m=+61.777756123" watchObservedRunningTime="2026-07-22 18:15:07.562983194 +0000 UTC m=+61.778630597"
Jul 22 18:15:07.575934 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:07.575894    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t" podStartSLOduration=0.876890813 podStartE2EDuration="9.575878983s" totalImagesPullingTime="8.69898817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:15:07.575167196 +0000 UTC m=+61.790814599" watchObservedRunningTime="2026-07-22 18:15:07.575878983 +0000 UTC m=+61.791526386"
Jul 22 18:15:11.933220 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:11.933175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:15:11.935631 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:11.935612    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 22 18:15:11.946312 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:11.946294    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 22 18:15:11.957345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:11.957328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvk4\" (UniqueName: \"kubernetes.io/projected/db97839c-0b2d-44c4-8fa1-aaaed691f651-kube-api-access-mhvk4\") pod \"network-check-target-nlzlv\" (UID: \"db97839c-0b2d-44c4-8fa1-aaaed691f651\") " pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:15:12.033434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.033412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:15:12.033552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.033440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:15:12.036015 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.035999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 22 18:15:12.036075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.036039    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 22 18:15:12.044160 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:12.044141    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 22 18:15:12.044238 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:12.044190    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:16:16.044176037 +0000 UTC m=+130.259823430 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : secret "metrics-daemon-secret" not found
Jul 22 18:15:12.046080 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.046062    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/60a152e6-60d1-4e08-86c5-8d03303c4cec-original-pull-secret\") pod \"global-pull-secret-syncer-2zn89\" (UID: \"60a152e6-60d1-4e08-86c5-8d03303c4cec\") " pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:15:12.234632 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.234564    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8sjds\""
Jul 22 18:15:12.242434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.242418    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:15:12.326497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.326464    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2zn89"
Jul 22 18:15:12.357666 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.357647    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlzlv"]
Jul 22 18:15:12.360265 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:15:12.360244    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb97839c_0b2d_44c4_8fa1_aaaed691f651.slice/crio-8d0bac3ed3f6e70d7c2b8edf8ac94381584b88e578bffbda5eae68a4bc212159 WatchSource:0}: Error finding container 8d0bac3ed3f6e70d7c2b8edf8ac94381584b88e578bffbda5eae68a4bc212159: Status 404 returned error can't find the container with id 8d0bac3ed3f6e70d7c2b8edf8ac94381584b88e578bffbda5eae68a4bc212159
Jul 22 18:15:12.445730 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.445586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2zn89"]
Jul 22 18:15:12.448082 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:15:12.448055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a152e6_60d1_4e08_86c5_8d03303c4cec.slice/crio-3ca110399eed3da90430fbec5747abb33bfc8c3a1b665ba69084018993e009ca WatchSource:0}: Error finding container 3ca110399eed3da90430fbec5747abb33bfc8c3a1b665ba69084018993e009ca: Status 404 returned error can't find the container with id 3ca110399eed3da90430fbec5747abb33bfc8c3a1b665ba69084018993e009ca
Jul 22 18:15:12.536954 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.536884    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:15:12.537080 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:12.537023    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:15:12.537122 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:12.537082    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:15:44.537065797 +0000 UTC m=+98.752713178 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:15:12.559848 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.559819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2zn89" event={"ID":"60a152e6-60d1-4e08-86c5-8d03303c4cec","Type":"ContainerStarted","Data":"3ca110399eed3da90430fbec5747abb33bfc8c3a1b665ba69084018993e009ca"}
Jul 22 18:15:12.560744 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:12.560724    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlzlv" event={"ID":"db97839c-0b2d-44c4-8fa1-aaaed691f651","Type":"ContainerStarted","Data":"8d0bac3ed3f6e70d7c2b8edf8ac94381584b88e578bffbda5eae68a4bc212159"}
Jul 22 18:15:16.574768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:16.574731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlzlv" event={"ID":"db97839c-0b2d-44c4-8fa1-aaaed691f651","Type":"ContainerStarted","Data":"1c24d5c6450eb23fef41d78444a842a896527ec325277ec2f3cd01897f78564f"}
Jul 22 18:15:16.575235 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:16.574850    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:15:16.590233 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:16.590176    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nlzlv" podStartSLOduration=67.278387976 podStartE2EDuration="1m10.590160584s" totalImagesPullingTime="3.311772608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:15:16.589898891 +0000 UTC m=+70.805546293" watchObservedRunningTime="2026-07-22 18:15:16.590160584 +0000 UTC m=+70.805807991"
Jul 22 18:15:17.579173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:17.579085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2zn89" event={"ID":"60a152e6-60d1-4e08-86c5-8d03303c4cec","Type":"ContainerStarted","Data":"90d608d2e83e6d7a4e66a88cbd6791c15821c01bb575ffc1c6c43b3c9122d8fb"}
Jul 22 18:15:17.593259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:17.593213    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2zn89" podStartSLOduration=65.796877571 podStartE2EDuration="1m10.593200149s" totalImagesPullingTime="4.796322578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:15:17.592645523 +0000 UTC m=+71.808292925" watchObservedRunningTime="2026-07-22 18:15:17.593200149 +0000 UTC m=+71.808847568"
Jul 22 18:15:44.555348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:44.555310    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:15:44.555743 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:44.555458    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:15:44.555743 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:15:44.555530    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:16:48.555514791 +0000 UTC m=+162.771162172 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:15:47.581801 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:15:47.581770    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nlzlv"
Jul 22 18:16:16.068398 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:16:16.068354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:16:16.068960 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:16.068485    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 22 18:16:16.068960 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:16.068569    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:18:18.068549284 +0000 UTC m=+252.284196681 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : secret "metrics-daemon-secret" not found
Jul 22 18:16:43.732051 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:43.732007    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bthdp" podUID="f06502df-814d-476e-9f23-9eaae5e32a1d"
Jul 22 18:16:43.814064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:16:43.814034    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:16:44.326504 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:44.326465    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:16:48.584063 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:16:48.584009    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:16:48.584484 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:48.584164    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 22 18:16:48.584484 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:16:48.584239    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls podName:f06502df-814d-476e-9f23-9eaae5e32a1d nodeName:}" failed. No retries permitted until 2026-07-22 18:18:50.584219508 +0000 UTC m=+284.799866907 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls") pod "dns-default-bthdp" (UID: "f06502df-814d-476e-9f23-9eaae5e32a1d") : secret "dns-default-metrics-tls" not found
Jul 22 18:16:57.314224 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:16:57.314189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:17:07.881868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:17:07.881834    2580 generic.go:356] "Generic (PLEG): container finished" podID="146b460e-963a-46c6-9325-3f35674e287d" containerID="1849d6f38b1a32eeed76e9f9970f204dee1068de3e15d4eec81684fc22221992" exitCode=255
Jul 22 18:17:07.882263 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:17:07.881902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t" event={"ID":"146b460e-963a-46c6-9325-3f35674e287d","Type":"ContainerDied","Data":"1849d6f38b1a32eeed76e9f9970f204dee1068de3e15d4eec81684fc22221992"}
Jul 22 18:17:07.882263 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:17:07.882203    2580 scope.go:117] "RemoveContainer" containerID="1849d6f38b1a32eeed76e9f9970f204dee1068de3e15d4eec81684fc22221992"
Jul 22 18:17:08.359189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:17:08.359151    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t"
Jul 22 18:17:08.886639 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:17:08.886604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-58496bdc46-cgf9t" event={"ID":"146b460e-963a-46c6-9325-3f35674e287d","Type":"ContainerStarted","Data":"8e2b284d90415640d6aa93959ccffcbcef291ee08cdd89a9092fc576a91f6318"}
Jul 22 18:18:07.268933 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.268895    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-svp7h"]
Jul 22 18:18:07.270859 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.270844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.273979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.273958    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d82fz\""
Jul 22 18:18:07.274101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.273995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 22 18:18:07.274537 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.274522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 22 18:18:07.274691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.274550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 22 18:18:07.274691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.274550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 22 18:18:07.288454 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.288428    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svp7h"]
Jul 22 18:18:07.423383 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.423344    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/26ddf080-af95-493c-96b0-802746aa037e-data-volume\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.423559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.423408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.423559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.423448    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.423559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.423492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9nn6\" (UniqueName: \"kubernetes.io/projected/26ddf080-af95-493c-96b0-802746aa037e-kube-api-access-k9nn6\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.423559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.423518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/26ddf080-af95-493c-96b0-802746aa037e-crio-socket\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524006    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9nn6\" (UniqueName: \"kubernetes.io/projected/26ddf080-af95-493c-96b0-802746aa037e-kube-api-access-k9nn6\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524064    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/26ddf080-af95-493c-96b0-802746aa037e-crio-socket\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/26ddf080-af95-493c-96b0-802746aa037e-data-volume\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524126    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524146    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524199    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/26ddf080-af95-493c-96b0-802746aa037e-crio-socket\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.524249 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:07.524234    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 18:18:07.524409 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:07.524274    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:07.524409 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:07.524313    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:08.024292394 +0000 UTC m=+242.239939794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : configmap "kube-rbac-proxy" not found
Jul 22 18:18:07.524409 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:07.524331    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:08.024322807 +0000 UTC m=+242.239970188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:07.524516 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.524451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/26ddf080-af95-493c-96b0-802746aa037e-data-volume\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:07.534618 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:07.534590    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9nn6\" (UniqueName: \"kubernetes.io/projected/26ddf080-af95-493c-96b0-802746aa037e-kube-api-access-k9nn6\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:08.027864 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:08.027826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:08.027864 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:08.027867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:08.028096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:08.027936    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 18:18:08.028096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:08.027992    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:08.028096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:08.028004    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:09.02798723 +0000 UTC m=+243.243634611 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : configmap "kube-rbac-proxy" not found
Jul 22 18:18:08.028096 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:08.028043    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:09.028028409 +0000 UTC m=+243.243675805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:09.036433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:09.036399    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:09.036864 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:09.036433    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:09.036864 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:09.036513    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 18:18:09.036864 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:09.036561    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:09.036864 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:09.036576    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:11.036561944 +0000 UTC m=+245.252209326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : configmap "kube-rbac-proxy" not found
Jul 22 18:18:09.036864 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:09.036605    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:11.036594284 +0000 UTC m=+245.252241664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:09.444698 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:09.444671    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jxvp4_16f3208c-6065-4712-9296-87de0e812da9/dns-node-resolver/0.log"
Jul 22 18:18:10.444937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:10.444910    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmgns_6ae127f4-a41c-4faa-8156-7ec3d2c46382/node-ca/0.log"
Jul 22 18:18:10.843291 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:10.843209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgxlm_107e48ae-5663-4d7d-9d6d-a26d5270144c/serve-healthcheck-canary/0.log"
Jul 22 18:18:11.049581 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:11.049552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:11.049581 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:11.049582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:11.049747 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:11.049673    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 18:18:11.049747 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:11.049686    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:11.049747 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:11.049738    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:15.049722053 +0000 UTC m=+249.265369453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : configmap "kube-rbac-proxy" not found
Jul 22 18:18:11.049921 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:11.049754    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:15.049748395 +0000 UTC m=+249.265395777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:15.079595 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:15.079557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:15.079595 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:15.079595    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:15.080136 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:15.079677    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 22 18:18:15.080136 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:15.079724    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:15.080136 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:15.079743    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:23.079725732 +0000 UTC m=+257.295373127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : configmap "kube-rbac-proxy" not found
Jul 22 18:18:15.080136 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:15.079761    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:23.079750274 +0000 UTC m=+257.295397659 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:18.103877 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:18.103828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:18:18.104345 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:18.103989    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 22 18:18:18.104345 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:18.104070    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs podName:3161c822-0d62-468b-9001-8ed2f861e17d nodeName:}" failed. No retries permitted until 2026-07-22 18:20:20.104048756 +0000 UTC m=+374.319696153 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs") pod "network-metrics-daemon-7qrxs" (UID: "3161c822-0d62-468b-9001-8ed2f861e17d") : secret "metrics-daemon-secret" not found
Jul 22 18:18:23.134949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:23.134915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:23.134949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:23.134951    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:23.135475 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:23.135074    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:23.135475 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:23.135141    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls podName:26ddf080-af95-493c-96b0-802746aa037e nodeName:}" failed. No retries permitted until 2026-07-22 18:18:39.135124148 +0000 UTC m=+273.350771530 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-svp7h" (UID: "26ddf080-af95-493c-96b0-802746aa037e") : secret "insights-runtime-extractor-tls" not found
Jul 22 18:18:23.135574 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:23.135552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/26ddf080-af95-493c-96b0-802746aa037e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:37.427070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.426994    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"]
Jul 22 18:18:37.429054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.429040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:37.431768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.431745    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 22 18:18:37.434595 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.434578    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rfrjs\""
Jul 22 18:18:37.439186 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.439163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"]
Jul 22 18:18:37.533196 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.533157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dj4vr\" (UID: \"86f2a168-648b-40f9-a1e3-ea8e337a3dc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:37.634412 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:37.634380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dj4vr\" (UID: \"86f2a168-648b-40f9-a1e3-ea8e337a3dc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:37.634579 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:37.634504    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 22 18:18:37.634579 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:37.634568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates podName:86f2a168-648b-40f9-a1e3-ea8e337a3dc8 nodeName:}" failed. No retries permitted until 2026-07-22 18:18:38.13455403 +0000 UTC m=+272.350201411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-dj4vr" (UID: "86f2a168-648b-40f9-a1e3-ea8e337a3dc8") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 22 18:18:38.137950 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:38.137918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dj4vr\" (UID: \"86f2a168-648b-40f9-a1e3-ea8e337a3dc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:38.140221 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:38.140191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/86f2a168-648b-40f9-a1e3-ea8e337a3dc8-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dj4vr\" (UID: \"86f2a168-648b-40f9-a1e3-ea8e337a3dc8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:38.337731 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:38.337704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:38.452309 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:38.452278    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"]
Jul 22 18:18:38.455330 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:38.455304    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f2a168_648b_40f9_a1e3_ea8e337a3dc8.slice/crio-6b6efdd9234aef3dfa0d5d241b74c6c466e3cb8c462c5c7aea4743a5692cf499 WatchSource:0}: Error finding container 6b6efdd9234aef3dfa0d5d241b74c6c466e3cb8c462c5c7aea4743a5692cf499: Status 404 returned error can't find the container with id 6b6efdd9234aef3dfa0d5d241b74c6c466e3cb8c462c5c7aea4743a5692cf499
Jul 22 18:18:39.132851 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:39.132800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr" event={"ID":"86f2a168-648b-40f9-a1e3-ea8e337a3dc8","Type":"ContainerStarted","Data":"6b6efdd9234aef3dfa0d5d241b74c6c466e3cb8c462c5c7aea4743a5692cf499"}
Jul 22 18:18:39.144107 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:39.144077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:39.146521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:39.146497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/26ddf080-af95-493c-96b0-802746aa037e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-svp7h\" (UID: \"26ddf080-af95-493c-96b0-802746aa037e\") " pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:39.379441 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:39.379401    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-svp7h"
Jul 22 18:18:39.596562 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:39.596526    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-svp7h"]
Jul 22 18:18:39.600105 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:39.600083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ddf080_af95_493c_96b0_802746aa037e.slice/crio-db2ac403680b60f50a7f74ea75d174e30c1a6bd5bbfa016abbf97808b58f03f3 WatchSource:0}: Error finding container db2ac403680b60f50a7f74ea75d174e30c1a6bd5bbfa016abbf97808b58f03f3: Status 404 returned error can't find the container with id db2ac403680b60f50a7f74ea75d174e30c1a6bd5bbfa016abbf97808b58f03f3
Jul 22 18:18:40.138051 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:40.137960    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svp7h" event={"ID":"26ddf080-af95-493c-96b0-802746aa037e","Type":"ContainerStarted","Data":"db2ac403680b60f50a7f74ea75d174e30c1a6bd5bbfa016abbf97808b58f03f3"}
Jul 22 18:18:40.140223 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:40.140162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr" event={"ID":"86f2a168-648b-40f9-a1e3-ea8e337a3dc8","Type":"ContainerStarted","Data":"08448805ab5a8ccf63b4888a13a20203a84376f4ee3f37490ef53780e21a739c"}
Jul 22 18:18:40.142168 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:40.140618    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:40.148269 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:40.148216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr"
Jul 22 18:18:40.160333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:40.159627    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dj4vr" podStartSLOduration=2.112285482 podStartE2EDuration="3.159609822s" totalImagesPullingTime="1.04732434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:40.159388743 +0000 UTC m=+274.375036147" watchObservedRunningTime="2026-07-22 18:18:40.159609822 +0000 UTC m=+274.375257228"
Jul 22 18:18:42.133782 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.133743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:18:42.144877 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.144846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.148285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148014    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 22 18:18:42.148285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148066    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 22 18:18:42.148285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzbg6\""
Jul 22 18:18:42.148285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:18:42.148285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148022    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 22 18:18:42.148737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 22 18:18:42.148737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 22 18:18:42.148737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148422    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 22 18:18:42.148917 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.148798    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 22 18:18:42.169031 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.169200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.169200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169142    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.169200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.169200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgjc\" (UniqueName: \"kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.169392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.169220    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270280 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270470 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270545 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgjc\" (UniqueName: \"kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270545 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270656 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.270719 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.270671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.271100 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.271052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.271368 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.271287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.271368 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.271316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.273530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.273506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.273530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.273518    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.279228 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.279202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgjc\" (UniqueName: \"kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc\") pod \"console-7884fc4877-cjw6p\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") " pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:42.457315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:42.457280    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:43.722564 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.722530    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:18:43.725043 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.725019    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.733472 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.733452    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 22 18:18:43.736123 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.736099    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:18:43.784443 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784617 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784472    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6k22\" (UniqueName: \"kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784617 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784617 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.784727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.784712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.885927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.885890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6k22\" (UniqueName: \"kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.885927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.885944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886175 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.885963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886172    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886342 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886389    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886824 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886788    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.886941 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.886919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.887073 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.887048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.887318 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.887291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.888750 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.888728    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.888850 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.888760    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:43.895677 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:43.895656    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6k22\" (UniqueName: \"kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22\") pod \"console-84d95589c-rrlgs\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") " pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:44.035837 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:44.035724    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:44.475604 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:44.475536    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:18:44.479468 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:44.479414    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa190188_7fe0_4254_b2cc_c4943509d7e2.slice/crio-bcf62d8411c1047cd5db1c4e60938dd555f3484dbddb0cd514ca1c8ca61e0772 WatchSource:0}: Error finding container bcf62d8411c1047cd5db1c4e60938dd555f3484dbddb0cd514ca1c8ca61e0772: Status 404 returned error can't find the container with id bcf62d8411c1047cd5db1c4e60938dd555f3484dbddb0cd514ca1c8ca61e0772
Jul 22 18:18:44.494134 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:44.494107    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:18:44.497037 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:44.497013    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98585563_6952_4cf2_bb77_7bf603c89971.slice/crio-981d89c1819e9bd3455d288c1b26114651dfc51b5da1acaadcd9f1ad33a87e3b WatchSource:0}: Error finding container 981d89c1819e9bd3455d288c1b26114651dfc51b5da1acaadcd9f1ad33a87e3b: Status 404 returned error can't find the container with id 981d89c1819e9bd3455d288c1b26114651dfc51b5da1acaadcd9f1ad33a87e3b
Jul 22 18:18:45.158959 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:45.158863    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svp7h" event={"ID":"26ddf080-af95-493c-96b0-802746aa037e","Type":"ContainerStarted","Data":"416d8e0f10f05c3eb1a52b32094e056981a600582f317f5a1f7e5a668ba45057"}
Jul 22 18:18:45.161229 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:45.161155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7884fc4877-cjw6p" event={"ID":"aa190188-7fe0-4254-b2cc-c4943509d7e2","Type":"ContainerStarted","Data":"bcf62d8411c1047cd5db1c4e60938dd555f3484dbddb0cd514ca1c8ca61e0772"}
Jul 22 18:18:45.163380 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:45.163330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d95589c-rrlgs" event={"ID":"98585563-6952-4cf2-bb77-7bf603c89971","Type":"ContainerStarted","Data":"981d89c1819e9bd3455d288c1b26114651dfc51b5da1acaadcd9f1ad33a87e3b"}
Jul 22 18:18:46.815079 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:46.815023    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-bthdp" podUID="f06502df-814d-476e-9f23-9eaae5e32a1d"
Jul 22 18:18:47.171612 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:47.171535    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:18:48.365916 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:48.365858    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" podUID="af85524e-af3f-4495-8caf-7f56ed552f68" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 18:18:49.857458 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.857423    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vwdbn"]
Jul 22 18:18:49.859724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.859703    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.862521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.862496    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 22 18:18:49.862647 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.862520    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 22 18:18:49.863219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.863202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 22 18:18:49.863219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.863210    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j2mzc\""
Jul 22 18:18:49.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.863232    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 22 18:18:49.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.863275    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 22 18:18:49.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.863330    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 22 18:18:49.936897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.936866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-metrics-client-ca\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.936897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.936902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.936924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-root\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.936954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-textfile\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.937056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s68w\" (UniqueName: \"kubernetes.io/projected/632f93a4-14f6-4feb-985a-d44f1510efc6-kube-api-access-7s68w\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.937094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-sys\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937221 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.937154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-wtmp\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937221 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.937175    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:49.937221 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:49.937195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038157 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-metrics-client-ca\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038157 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-root\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-textfile\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s68w\" (UniqueName: \"kubernetes.io/projected/632f93a4-14f6-4feb-985a-d44f1510efc6-kube-api-access-7s68w\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-root\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038338    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-sys\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-sys\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-wtmp\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038737 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:50.038552    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 22 18:18:50.038737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038595    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-wtmp\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038737 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:18:50.038620    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls podName:632f93a4-14f6-4feb-985a-d44f1510efc6 nodeName:}" failed. No retries permitted until 2026-07-22 18:18:50.538601705 +0000 UTC m=+284.754249088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls") pod "node-exporter-vwdbn" (UID: "632f93a4-14f6-4feb-985a-d44f1510efc6") : secret "node-exporter-tls" not found
Jul 22 18:18:50.038737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038668    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-textfile\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038983 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-metrics-client-ca\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.038983 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.038907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.040662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.040644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.047072 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.047048    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s68w\" (UniqueName: \"kubernetes.io/projected/632f93a4-14f6-4feb-985a-d44f1510efc6-kube-api-access-7s68w\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.183339 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.183301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svp7h" event={"ID":"26ddf080-af95-493c-96b0-802746aa037e","Type":"ContainerStarted","Data":"18ddda75f209b69ccedc8e73b924b2678d059479c77982587d2fbbc10893166e"}
Jul 22 18:18:50.184791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.184765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7884fc4877-cjw6p" event={"ID":"aa190188-7fe0-4254-b2cc-c4943509d7e2","Type":"ContainerStarted","Data":"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"}
Jul 22 18:18:50.186103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.186084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d95589c-rrlgs" event={"ID":"98585563-6952-4cf2-bb77-7bf603c89971","Type":"ContainerStarted","Data":"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"}
Jul 22 18:18:50.207611 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.207572    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7884fc4877-cjw6p" podStartSLOduration=3.549961682 podStartE2EDuration="8.207557718s" totalImagesPullingTime="4.657596036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:50.206686081 +0000 UTC m=+284.422333514" watchObservedRunningTime="2026-07-22 18:18:50.207557718 +0000 UTC m=+284.423205099"
Jul 22 18:18:50.224335 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.224292    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84d95589c-rrlgs" podStartSLOduration=2.599335699 podStartE2EDuration="7.224280364s" totalImagesPullingTime="4.624944665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:18:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:50.222936353 +0000 UTC m=+284.438583757" watchObservedRunningTime="2026-07-22 18:18:50.224280364 +0000 UTC m=+284.439927766"
Jul 22 18:18:50.542295 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.542202    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.544414 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.544397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/632f93a4-14f6-4feb-985a-d44f1510efc6-node-exporter-tls\") pod \"node-exporter-vwdbn\" (UID: \"632f93a4-14f6-4feb-985a-d44f1510efc6\") " pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.643514 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.643476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:18:50.645646 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.645629    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f06502df-814d-476e-9f23-9eaae5e32a1d-metrics-tls\") pod \"dns-default-bthdp\" (UID: \"f06502df-814d-476e-9f23-9eaae5e32a1d\") " pod="openshift-dns/dns-default-bthdp"
Jul 22 18:18:50.768388 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.768362    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vwdbn"
Jul 22 18:18:50.774332 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.774308    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ttgch\""
Jul 22 18:18:50.778217 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:50.778194    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632f93a4_14f6_4feb_985a_d44f1510efc6.slice/crio-1493a6f6665750a5093dc711ef2b9f1b14d46bdba7d4a95c1640bb028278cda0 WatchSource:0}: Error finding container 1493a6f6665750a5093dc711ef2b9f1b14d46bdba7d4a95c1640bb028278cda0: Status 404 returned error can't find the container with id 1493a6f6665750a5093dc711ef2b9f1b14d46bdba7d4a95c1640bb028278cda0
Jul 22 18:18:50.782425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.782408    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:18:50.908739 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:50.908700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bthdp"]
Jul 22 18:18:50.913931 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:50.913903    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06502df_814d_476e_9f23_9eaae5e32a1d.slice/crio-58b9dffcfd08dafc321260e7f859fa0d0c9c79da0b8bb10f3a6b883df2f54378 WatchSource:0}: Error finding container 58b9dffcfd08dafc321260e7f859fa0d0c9c79da0b8bb10f3a6b883df2f54378: Status 404 returned error can't find the container with id 58b9dffcfd08dafc321260e7f859fa0d0c9c79da0b8bb10f3a6b883df2f54378
Jul 22 18:18:51.191525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:51.191472    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bthdp" event={"ID":"f06502df-814d-476e-9f23-9eaae5e32a1d","Type":"ContainerStarted","Data":"58b9dffcfd08dafc321260e7f859fa0d0c9c79da0b8bb10f3a6b883df2f54378"}
Jul 22 18:18:51.192866 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:51.192831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vwdbn" event={"ID":"632f93a4-14f6-4feb-985a-d44f1510efc6","Type":"ContainerStarted","Data":"1493a6f6665750a5093dc711ef2b9f1b14d46bdba7d4a95c1640bb028278cda0"}
Jul 22 18:18:52.198738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.198699    2580 generic.go:356] "Generic (PLEG): container finished" podID="632f93a4-14f6-4feb-985a-d44f1510efc6" containerID="d2e1a00922ab7624415ce414f695869fd27e62921a0be86c628565d96b169985" exitCode=0
Jul 22 18:18:52.199227 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.198777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vwdbn" event={"ID":"632f93a4-14f6-4feb-985a-d44f1510efc6","Type":"ContainerDied","Data":"d2e1a00922ab7624415ce414f695869fd27e62921a0be86c628565d96b169985"}
Jul 22 18:18:52.457935 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.457840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:52.457935 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.457920    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:52.464182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.464152    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:52.805006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.804923    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bfff888c-k4cbx"]
Jul 22 18:18:52.807847 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.807825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.810623 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.810598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tr2x6\""
Jul 22 18:18:52.810763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.810630    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 22 18:18:52.810763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.810641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 22 18:18:52.810763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.810694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 22 18:18:52.810763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.810610    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 22 18:18:52.811028 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.811007    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 22 18:18:52.811400 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.811372    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fni86efc58dpp\""
Jul 22 18:18:52.819970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.819950    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bfff888c-k4cbx"]
Jul 22 18:18:52.863207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dv9f\" (UniqueName: \"kubernetes.io/projected/43165952-70c1-46cd-8180-22828d6af0c8-kube-api-access-7dv9f\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863310    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863377 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-grpc-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863599 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43165952-70c1-46cd-8180-22828d6af0c8-metrics-client-ca\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863599 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.863599 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.863504    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964516 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dv9f\" (UniqueName: \"kubernetes.io/projected/43165952-70c1-46cd-8180-22828d6af0c8-kube-api-access-7dv9f\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964516 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964544    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-grpc-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43165952-70c1-46cd-8180-22828d6af0c8-metrics-client-ca\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.964761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.964743    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.965750 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.965712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43165952-70c1-46cd-8180-22828d6af0c8-metrics-client-ca\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.967698 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.967600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.967952 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.967934    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.968028 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.968000    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.968160 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.968136    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.968265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.968188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-grpc-tls\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.968265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.968190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/43165952-70c1-46cd-8180-22828d6af0c8-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:52.972860 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:52.972842    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dv9f\" (UniqueName: \"kubernetes.io/projected/43165952-70c1-46cd-8180-22828d6af0c8-kube-api-access-7dv9f\") pod \"thanos-querier-7bfff888c-k4cbx\" (UID: \"43165952-70c1-46cd-8180-22828d6af0c8\") " pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:53.118671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:53.118589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:53.205757 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:53.205703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vwdbn" event={"ID":"632f93a4-14f6-4feb-985a-d44f1510efc6","Type":"ContainerStarted","Data":"71c42a37833011f69786e283928a441bddac682c70dbc3232b2fb8ba4d56acef"}
Jul 22 18:18:53.206392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:53.205773    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vwdbn" event={"ID":"632f93a4-14f6-4feb-985a-d44f1510efc6","Type":"ContainerStarted","Data":"f443c69790b43e331249bb20ba9440a4da4a6324667e937af0bf330e7266baab"}
Jul 22 18:18:53.210242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:53.210220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:18:53.226667 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:53.226621    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vwdbn" podStartSLOduration=3.050846047 podStartE2EDuration="4.226603976s" totalImagesPullingTime="1.096136142s" totalInitContainerRuntime="79.621787ms" isStatefulPod=true podCreationTimestamp="2026-07-22 18:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:53.225479655 +0000 UTC m=+287.441127071" watchObservedRunningTime="2026-07-22 18:18:53.226603976 +0000 UTC m=+287.442251417"
Jul 22 18:18:54.035998 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.035961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:54.036201 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.036032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:54.041206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.041183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:54.213038 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.213010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:18:54.254235 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.254204    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:18:54.902094 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:54.901990    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bfff888c-k4cbx"]
Jul 22 18:18:54.904466 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:18:54.904430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43165952_70c1_46cd_8180_22828d6af0c8.slice/crio-325afc493e26ebb06b08142c5643b9241d6091f0c96bfbee0271704c61f80e52 WatchSource:0}: Error finding container 325afc493e26ebb06b08142c5643b9241d6091f0c96bfbee0271704c61f80e52: Status 404 returned error can't find the container with id 325afc493e26ebb06b08142c5643b9241d6091f0c96bfbee0271704c61f80e52
Jul 22 18:18:55.213203 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.213166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"325afc493e26ebb06b08142c5643b9241d6091f0c96bfbee0271704c61f80e52"}
Jul 22 18:18:55.215149 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.215123    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-svp7h" event={"ID":"26ddf080-af95-493c-96b0-802746aa037e","Type":"ContainerStarted","Data":"223ca133aed8359a2403fb0ff24d7ac198edf8259aa7616d475d9e706ca1de58"}
Jul 22 18:18:55.216887 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.216865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bthdp" event={"ID":"f06502df-814d-476e-9f23-9eaae5e32a1d","Type":"ContainerStarted","Data":"a4a19ab0da144432a924b7f9bfe8ac5afffabbdc94f2889510ed886c25654979"}
Jul 22 18:18:55.217014 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.216903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bthdp" event={"ID":"f06502df-814d-476e-9f23-9eaae5e32a1d","Type":"ContainerStarted","Data":"f7bb12c0f19c6ce7a77a48b5df88e9d5018a16263142bdb5d895633bedd632ce"}
Jul 22 18:18:55.217014 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.216999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:18:55.231398 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.231182    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-svp7h" podStartSLOduration=33.228185563 podStartE2EDuration="48.23115772s" totalImagesPullingTime="15.002972157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 18:18:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:55.230975634 +0000 UTC m=+289.446623038" watchObservedRunningTime="2026-07-22 18:18:55.23115772 +0000 UTC m=+289.446805125"
Jul 22 18:18:55.246277 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:55.246230    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bthdp" podStartSLOduration=251.41819277 podStartE2EDuration="4m15.246214347s" totalImagesPullingTime="3.828021577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:55.246003735 +0000 UTC m=+289.461651138" watchObservedRunningTime="2026-07-22 18:18:55.246214347 +0000 UTC m=+289.461861743"
Jul 22 18:18:57.227542 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:57.227506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"a29c09151ef5c6081c68a6cb7ac48af13a0c410732ebf8772a6df08ae9a49da0"}
Jul 22 18:18:57.227979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:57.227590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"2ba2c4fa26a8d4374422cbf074762c72ef93bc068e6d9d5f3ebadf245ea5ac46"}
Jul 22 18:18:57.227979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:57.227634    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"951d69a2209b7be0a8a817e7879d513520d20d6642361503eba4800ccacd8d44"}
Jul 22 18:18:58.233430 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.233396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"6f83e6e6e9f17526ae36cb6c2946a1cf89af25c9c4443c8f7fb77595de633c86"}
Jul 22 18:18:58.233800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.233482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"537dd4d995d6b0380af0a24c0f67f55367ed7aba186cd0063344adc68d3c255b"}
Jul 22 18:18:58.233800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.233509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" event={"ID":"43165952-70c1-46cd-8180-22828d6af0c8","Type":"ContainerStarted","Data":"26f68e67eff175873d7598e51e17c4e203544ccd2c64a2caa0cc36bd80b98f0a"}
Jul 22 18:18:58.233800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.233624    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:18:58.257923 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.257855    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx" podStartSLOduration=3.898970287 podStartE2EDuration="6.257840247s" totalImagesPullingTime="2.35886996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:18:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:18:58.257012477 +0000 UTC m=+292.472659887" watchObservedRunningTime="2026-07-22 18:18:58.257840247 +0000 UTC m=+292.473487628"
Jul 22 18:18:58.366272 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:18:58.366235    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" podUID="af85524e-af3f-4495-8caf-7f56ed552f68" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 18:19:00.315500 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:19:00.315458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7qrxs" podUID="3161c822-0d62-468b-9001-8ed2f861e17d"
Jul 22 18:19:04.242749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:04.242719    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bfff888c-k4cbx"
Jul 22 18:19:05.222781 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:05.222748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bthdp"
Jul 22 18:19:06.154606 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:06.154579    2580 kubelet.go:1648] "Image garbage collection succeeded"
Jul 22 18:19:08.366608 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:08.366570    2580 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" podUID="af85524e-af3f-4495-8caf-7f56ed552f68" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 22 18:19:08.368949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:08.366636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn"
Jul 22 18:19:08.368949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:08.367096    2580 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"6abae5d1233a16dfed8bde1eae4c87e52460d4b9d15a9a857a41966282d94fd5"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 22 18:19:08.368949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:08.367146    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" podUID="af85524e-af3f-4495-8caf-7f56ed552f68" containerName="service-proxy" containerID="cri-o://6abae5d1233a16dfed8bde1eae4c87e52460d4b9d15a9a857a41966282d94fd5" gracePeriod=30
Jul 22 18:19:09.272475 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:09.272441    2580 generic.go:356] "Generic (PLEG): container finished" podID="af85524e-af3f-4495-8caf-7f56ed552f68" containerID="6abae5d1233a16dfed8bde1eae4c87e52460d4b9d15a9a857a41966282d94fd5" exitCode=2
Jul 22 18:19:09.272641 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:09.272513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerDied","Data":"6abae5d1233a16dfed8bde1eae4c87e52460d4b9d15a9a857a41966282d94fd5"}
Jul 22 18:19:09.272641 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:09.272558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78fb5558fc-tf5hn" event={"ID":"af85524e-af3f-4495-8caf-7f56ed552f68","Type":"ContainerStarted","Data":"6600bc5636a2d4b203245e2659ef54f1c99e9c56b93bbe6ed1ae330a4d98e18a"}
Jul 22 18:19:12.279623 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:12.279589    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:19:13.314835 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:13.314770    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:19:20.236796 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:20.236733    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7884fc4877-cjw6p" podUID="aa190188-7fe0-4254-b2cc-c4943509d7e2" containerName="console" containerID="cri-o://d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef" gracePeriod=15
Jul 22 18:19:21.060952 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.060924    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:19:21.102136 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102307 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgjc\" (UniqueName: \"kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc\") pod \"aa190188-7fe0-4254-b2cc-c4943509d7e2\" (UID: \"aa190188-7fe0-4254-b2cc-c4943509d7e2\") "
Jul 22 18:19:21.102721 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102676    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca" (OuterVolumeSpecName: "service-ca") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:21.102721 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:21.102721 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.102713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config" (OuterVolumeSpecName: "console-config") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:21.104412 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.104392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:19:21.104480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.104441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:19:21.104518 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.104468    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc" (OuterVolumeSpecName: "kube-api-access-5jgjc") pod "aa190188-7fe0-4254-b2cc-c4943509d7e2" (UID: "aa190188-7fe0-4254-b2cc-c4943509d7e2"). InnerVolumeSpecName "kube-api-access-5jgjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:19:21.202958 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202921    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-oauth-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.202958 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202952    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.202958 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202961    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgjc\" (UniqueName: \"kubernetes.io/projected/aa190188-7fe0-4254-b2cc-c4943509d7e2-kube-api-access-5jgjc\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.203177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202970    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-oauth-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.203177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202978    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa190188-7fe0-4254-b2cc-c4943509d7e2-console-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.203177 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.202986    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa190188-7fe0-4254-b2cc-c4943509d7e2-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:21.313565 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.313529    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa190188-7fe0-4254-b2cc-c4943509d7e2" containerID="d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef" exitCode=0
Jul 22 18:19:21.313967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.313576    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7884fc4877-cjw6p"
Jul 22 18:19:21.313967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.313612    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7884fc4877-cjw6p" event={"ID":"aa190188-7fe0-4254-b2cc-c4943509d7e2","Type":"ContainerDied","Data":"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"}
Jul 22 18:19:21.313967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.313665    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7884fc4877-cjw6p" event={"ID":"aa190188-7fe0-4254-b2cc-c4943509d7e2","Type":"ContainerDied","Data":"bcf62d8411c1047cd5db1c4e60938dd555f3484dbddb0cd514ca1c8ca61e0772"}
Jul 22 18:19:21.313967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.313686    2580 scope.go:117] "RemoveContainer" containerID="d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"
Jul 22 18:19:21.321848 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.321827    2580 scope.go:117] "RemoveContainer" containerID="d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"
Jul 22 18:19:21.322095 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:19:21.322075    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef\": container with ID starting with d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef not found: ID does not exist" containerID="d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"
Jul 22 18:19:21.322162 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.322107    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef"} err="failed to get container status \"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef\": rpc error: code = NotFound desc = could not find container \"d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef\": container with ID starting with d581fca2d3cc18bf86999003809ee883edc1ce69a887e951f0e685950f4783ef not found: ID does not exist"
Jul 22 18:19:21.333178 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.333145    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:19:21.337121 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:21.337101    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7884fc4877-cjw6p"]
Jul 22 18:19:22.318301 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:22.318266    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa190188-7fe0-4254-b2cc-c4943509d7e2" path="/var/lib/kubelet/pods/aa190188-7fe0-4254-b2cc-c4943509d7e2/volumes"
Jul 22 18:19:37.299660 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.299602    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84d95589c-rrlgs" podUID="98585563-6952-4cf2-bb77-7bf603c89971" containerName="console" containerID="cri-o://8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7" gracePeriod=15
Jul 22 18:19:37.531173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.531149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:19:37.621740 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621664    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621740 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621743    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621837    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621867    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6k22\" (UniqueName: \"kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.621957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.621900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config\") pod \"98585563-6952-4cf2-bb77-7bf603c89971\" (UID: \"98585563-6952-4cf2-bb77-7bf603c89971\") "
Jul 22 18:19:37.622274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.622226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:37.622274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.622244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca" (OuterVolumeSpecName: "service-ca") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:37.622274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.622236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config" (OuterVolumeSpecName: "console-config") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:37.622274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.622257    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:19:37.623820 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.623783    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:19:37.623996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.623975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22" (OuterVolumeSpecName: "kube-api-access-j6k22") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "kube-api-access-j6k22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:19:37.624058 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.624005    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "98585563-6952-4cf2-bb77-7bf603c89971" (UID: "98585563-6952-4cf2-bb77-7bf603c89971"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:19:37.722672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722618    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-oauth-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722663    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/98585563-6952-4cf2-bb77-7bf603c89971-console-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722677    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722690    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-oauth-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722702    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-console-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722713    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98585563-6952-4cf2-bb77-7bf603c89971-trusted-ca-bundle\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:37.722965 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:37.722724    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6k22\" (UniqueName: \"kubernetes.io/projected/98585563-6952-4cf2-bb77-7bf603c89971-kube-api-access-j6k22\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:19:38.372566 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.372533    2580 generic.go:356] "Generic (PLEG): container finished" podID="98585563-6952-4cf2-bb77-7bf603c89971" containerID="8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7" exitCode=0
Jul 22 18:19:38.373006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.372593    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84d95589c-rrlgs"
Jul 22 18:19:38.373006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.372611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d95589c-rrlgs" event={"ID":"98585563-6952-4cf2-bb77-7bf603c89971","Type":"ContainerDied","Data":"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"}
Jul 22 18:19:38.373006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.372663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84d95589c-rrlgs" event={"ID":"98585563-6952-4cf2-bb77-7bf603c89971","Type":"ContainerDied","Data":"981d89c1819e9bd3455d288c1b26114651dfc51b5da1acaadcd9f1ad33a87e3b"}
Jul 22 18:19:38.373006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.372683    2580 scope.go:117] "RemoveContainer" containerID="8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"
Jul 22 18:19:38.381472 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.381448    2580 scope.go:117] "RemoveContainer" containerID="8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"
Jul 22 18:19:38.381734 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:19:38.381716    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7\": container with ID starting with 8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7 not found: ID does not exist" containerID="8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"
Jul 22 18:19:38.381837 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.381743    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7"} err="failed to get container status \"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7\": rpc error: code = NotFound desc = could not find container \"8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7\": container with ID starting with 8d0ad690377fcb86ffafdcb3edb9e78a9795ed2eb46f1b6645959831f98cc0c7 not found: ID does not exist"
Jul 22 18:19:38.389305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.389278    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:19:38.393396 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:38.393374    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84d95589c-rrlgs"]
Jul 22 18:19:40.321857 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:19:40.321799    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98585563-6952-4cf2-bb77-7bf603c89971" path="/var/lib/kubelet/pods/98585563-6952-4cf2-bb77-7bf603c89971/volumes"
Jul 22 18:20:20.159121 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.159079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:20:20.161375 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.161351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3161c822-0d62-468b-9001-8ed2f861e17d-metrics-certs\") pod \"network-metrics-daemon-7qrxs\" (UID: \"3161c822-0d62-468b-9001-8ed2f861e17d\") " pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:20:20.218075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.218052    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vjc8f\""
Jul 22 18:20:20.225651 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.225627    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qrxs"
Jul 22 18:20:20.346049 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.346018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qrxs"]
Jul 22 18:20:20.348686 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:20:20.348659    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3161c822_0d62_468b_9001_8ed2f861e17d.slice/crio-4f235ddb7b142dfd8ad20504ad37bf8e49df5c514a3fb4b54045e609c884908b WatchSource:0}: Error finding container 4f235ddb7b142dfd8ad20504ad37bf8e49df5c514a3fb4b54045e609c884908b: Status 404 returned error can't find the container with id 4f235ddb7b142dfd8ad20504ad37bf8e49df5c514a3fb4b54045e609c884908b
Jul 22 18:20:20.350509 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.350495    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:20:20.515526 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:20.515488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qrxs" event={"ID":"3161c822-0d62-468b-9001-8ed2f861e17d","Type":"ContainerStarted","Data":"4f235ddb7b142dfd8ad20504ad37bf8e49df5c514a3fb4b54045e609c884908b"}
Jul 22 18:20:21.520343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:21.520296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qrxs" event={"ID":"3161c822-0d62-468b-9001-8ed2f861e17d","Type":"ContainerStarted","Data":"689e0426388f99fc5616bf482db43c7c4839fbba2f31b4e42cc9dc4548897d8f"}
Jul 22 18:20:21.520749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:21.520364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qrxs" event={"ID":"3161c822-0d62-468b-9001-8ed2f861e17d","Type":"ContainerStarted","Data":"e258d5cebffd55aa1814d9d5c31720b82e14eabc64617d9347395f5831ec488f"}
Jul 22 18:20:21.537993 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:21.537946    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qrxs" podStartSLOduration=373.757825488 podStartE2EDuration="6m14.537929683s" totalImagesPullingTime="780.104195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:20:21.536688466 +0000 UTC m=+375.752335870" watchObservedRunningTime="2026-07-22 18:20:21.537929683 +0000 UTC m=+375.753577080"
Jul 22 18:20:22.115322 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115292    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:20:22.115547 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115536    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa190188-7fe0-4254-b2cc-c4943509d7e2" containerName="console"
Jul 22 18:20:22.115598 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115549    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa190188-7fe0-4254-b2cc-c4943509d7e2" containerName="console"
Jul 22 18:20:22.115598 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115561    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98585563-6952-4cf2-bb77-7bf603c89971" containerName="console"
Jul 22 18:20:22.115598 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115567    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98585563-6952-4cf2-bb77-7bf603c89971" containerName="console"
Jul 22 18:20:22.115685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115618    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa190188-7fe0-4254-b2cc-c4943509d7e2" containerName="console"
Jul 22 18:20:22.115685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.115628    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98585563-6952-4cf2-bb77-7bf603c89971" containerName="console"
Jul 22 18:20:22.117424 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.117409    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.120001 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.119980    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 22 18:20:22.120252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.120236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 22 18:20:22.120298 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.120237    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 22 18:20:22.120860 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.120846    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 22 18:20:22.121091 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.121078    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 22 18:20:22.121137 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.121095    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kzbg6\""
Jul 22 18:20:22.121429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.121411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 22 18:20:22.121493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.121429    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 22 18:20:22.125434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.125416    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 22 18:20:22.134239 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.134218    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:20:22.274055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274020    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fj8\" (UniqueName: \"kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274193    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274213    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.274274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.274271    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375080    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375154    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fj8\" (UniqueName: \"kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375285    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.375940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375920    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.376087 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.375914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.376419 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.376397    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.377594 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.377574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.377832 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.377793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.384001 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.383981    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fj8\" (UniqueName: \"kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8\") pod \"console-74f47d4cdf-9zbpb\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") " pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.426082 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.426058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:22.547218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:22.547180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:20:22.549734 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:20:22.549707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cab0f3f_5465_4c16_9193_e0e5acc1b2dd.slice/crio-722bc42d51436e892d31dd35b3c607880a726272e3ad60c7a0ed707fb9baa3a4 WatchSource:0}: Error finding container 722bc42d51436e892d31dd35b3c607880a726272e3ad60c7a0ed707fb9baa3a4: Status 404 returned error can't find the container with id 722bc42d51436e892d31dd35b3c607880a726272e3ad60c7a0ed707fb9baa3a4
Jul 22 18:20:23.528866 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:23.528831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f47d4cdf-9zbpb" event={"ID":"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd","Type":"ContainerStarted","Data":"d5aaf69491833e9123ada0faa4724f10beccafc30e3224fc9424a62d915604b2"}
Jul 22 18:20:23.529041 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:23.528881    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f47d4cdf-9zbpb" event={"ID":"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd","Type":"ContainerStarted","Data":"722bc42d51436e892d31dd35b3c607880a726272e3ad60c7a0ed707fb9baa3a4"}
Jul 22 18:20:23.545388 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:23.545340    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74f47d4cdf-9zbpb" podStartSLOduration=1.5453215930000002 podStartE2EDuration="1.545321593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:20:23.544844625 +0000 UTC m=+377.760492030" watchObservedRunningTime="2026-07-22 18:20:23.545321593 +0000 UTC m=+377.760968990"
Jul 22 18:20:32.427122 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:32.427093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:32.427531 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:32.427168    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:32.431857 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:32.431839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:20:32.568783 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:20:32.568756    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:21:48.079395 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.075862    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jshg"]
Jul 22 18:21:48.079395 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.078999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.081626 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.081605    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-spxpc\""
Jul 22 18:21:48.081754 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.081639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 22 18:21:48.081754 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.081647    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 22 18:21:48.086787 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.086764    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jshg"]
Jul 22 18:21:48.172751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.172721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltwj6\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-kube-api-access-ltwj6\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.172937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.172769    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.273662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.273629    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltwj6\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-kube-api-access-ltwj6\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.273769 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.273683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.282667 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.282637    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.282831 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.282797    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltwj6\" (UniqueName: \"kubernetes.io/projected/fa57266b-c305-446d-9e37-381b66b20f46-kube-api-access-ltwj6\") pod \"cert-manager-webhook-6486ffd997-6jshg\" (UID: \"fa57266b-c305-446d-9e37-381b66b20f46\") " pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.388373 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.388286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:48.508733 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.508703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-6jshg"]
Jul 22 18:21:48.511975 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:21:48.511935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa57266b_c305_446d_9e37_381b66b20f46.slice/crio-64129ee15967b2d55a6d587b0f20395e7e8aa661dd425eb8ddde4474e6654bd3 WatchSource:0}: Error finding container 64129ee15967b2d55a6d587b0f20395e7e8aa661dd425eb8ddde4474e6654bd3: Status 404 returned error can't find the container with id 64129ee15967b2d55a6d587b0f20395e7e8aa661dd425eb8ddde4474e6654bd3
Jul 22 18:21:48.816579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:48.816544    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg" event={"ID":"fa57266b-c305-446d-9e37-381b66b20f46","Type":"ContainerStarted","Data":"64129ee15967b2d55a6d587b0f20395e7e8aa661dd425eb8ddde4474e6654bd3"}
Jul 22 18:21:51.829504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:51.829417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg" event={"ID":"fa57266b-c305-446d-9e37-381b66b20f46","Type":"ContainerStarted","Data":"82321a0bc66ddda7d654ca808b7e168de7ce6c2128a8a31ff8a864a9f61d5933"}
Jul 22 18:21:51.829504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:51.829477    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:21:51.847607 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:51.847551    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg" podStartSLOduration=0.820811759 podStartE2EDuration="3.847535723s" totalImagesPullingTime="3.026723964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:21:51.846760388 +0000 UTC m=+466.062407792" watchObservedRunningTime="2026-07-22 18:21:51.847535723 +0000 UTC m=+466.063183125"
Jul 22 18:21:57.836142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:21:57.836106    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-6jshg"
Jul 22 18:22:23.330267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.330233    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"]
Jul 22 18:22:23.335799 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.335779    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.338315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.338288    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 22 18:22:23.338315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.338299    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 22 18:22:23.338480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.338317    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 22 18:22:23.338480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.338407    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 22 18:22:23.338480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.338432    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wz8qr\""
Jul 22 18:22:23.349571 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.349549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"]
Jul 22 18:22:23.431839 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.431784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-tmp\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.432016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.431872    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqf65\" (UniqueName: \"kubernetes.io/projected/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-kube-api-access-vqf65\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.432016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.431902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.432016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.431924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-manifests\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.432016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.432006    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.532857 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.532796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqf65\" (UniqueName: \"kubernetes.io/projected/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-kube-api-access-vqf65\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.533037 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.532865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.533037 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.532891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-manifests\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.533037 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.532919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.533037 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.532954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-tmp\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.533409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.533381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-manifests\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.535299 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.535274    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-tmp\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.535418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.535371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-webhook-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.535418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.535369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.540607 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.540585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqf65\" (UniqueName: \"kubernetes.io/projected/5fd4abd6-c7f6-4cd1-b258-656ad70048fd-kube-api-access-vqf65\") pod \"opendatahub-operator-controller-manager-549865fcdf-pqbqc\" (UID: \"5fd4abd6-c7f6-4cd1-b258-656ad70048fd\") " pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.645880 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.645776    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:23.777013 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.776981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"]
Jul 22 18:22:23.779856 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:23.779788    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd4abd6_c7f6_4cd1_b258_656ad70048fd.slice/crio-83023f1903531af88fbeccdb6e234386c37340441b606d7f54ceaa3bdfbe261b WatchSource:0}: Error finding container 83023f1903531af88fbeccdb6e234386c37340441b606d7f54ceaa3bdfbe261b: Status 404 returned error can't find the container with id 83023f1903531af88fbeccdb6e234386c37340441b606d7f54ceaa3bdfbe261b
Jul 22 18:22:23.934981 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:23.934947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc" event={"ID":"5fd4abd6-c7f6-4cd1-b258-656ad70048fd","Type":"ContainerStarted","Data":"83023f1903531af88fbeccdb6e234386c37340441b606d7f54ceaa3bdfbe261b"}
Jul 22 18:22:26.948344 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:26.948309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc" event={"ID":"5fd4abd6-c7f6-4cd1-b258-656ad70048fd","Type":"ContainerStarted","Data":"1454a5afeae2cf9fbdbc2704eff04e64c6db527ea7a12452c971e90cf12b08f9"}
Jul 22 18:22:27.953948 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:27.953920    2580 generic.go:356] "Generic (PLEG): container finished" podID="5fd4abd6-c7f6-4cd1-b258-656ad70048fd" containerID="1454a5afeae2cf9fbdbc2704eff04e64c6db527ea7a12452c971e90cf12b08f9" exitCode=0
Jul 22 18:22:27.954323 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:27.953963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc" event={"ID":"5fd4abd6-c7f6-4cd1-b258-656ad70048fd","Type":"ContainerDied","Data":"1454a5afeae2cf9fbdbc2704eff04e64c6db527ea7a12452c971e90cf12b08f9"}
Jul 22 18:22:28.960330 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:28.960296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc" event={"ID":"5fd4abd6-c7f6-4cd1-b258-656ad70048fd","Type":"ContainerStarted","Data":"707bb09948e205c4cca8fdf1d9ed05def4b93ffde6ddf080dfa93c3cec63c25e"}
Jul 22 18:22:28.960789 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:28.960452    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:28.984020 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:28.983943    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc" podStartSLOduration=1.492298319 podStartE2EDuration="5.983927666s" totalImagesPullingTime="3.474387382s" totalInitContainerRuntime="1.017241965s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:28.982671143 +0000 UTC m=+503.198318556" watchObservedRunningTime="2026-07-22 18:22:28.983927666 +0000 UTC m=+503.199575068"
Jul 22 18:22:39.966460 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:39.966427    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-549865fcdf-pqbqc"
Jul 22 18:22:43.591563 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.591525    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"]
Jul 22 18:22:43.601218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.601195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.603412 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.603390    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 22 18:22:43.604245 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.604221    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 22 18:22:43.604245 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.604240    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 22 18:22:43.604406 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.604243    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 22 18:22:43.610713 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.610691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"]
Jul 22 18:22:43.689214 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.689183    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd475\" (UniqueName: \"kubernetes.io/projected/82946fd3-0ef7-4f51-8760-85524d2f27e5-kube-api-access-sd475\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.689380 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.689221    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82946fd3-0ef7-4f51-8760-85524d2f27e5-tmp\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.689380 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.689277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82946fd3-0ef7-4f51-8760-85524d2f27e5-tls-certs\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.737169 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.737134    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sfjkm"]
Jul 22 18:22:43.739305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.739287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.741528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.741508    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 22 18:22:43.741782 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.741768    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8zq44\""
Jul 22 18:22:43.755128 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.755102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sfjkm"]
Jul 22 18:22:43.790098 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.790057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82946fd3-0ef7-4f51-8760-85524d2f27e5-tmp\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.790264 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.790138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82946fd3-0ef7-4f51-8760-85524d2f27e5-tls-certs\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.790264 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.790187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6p5\" (UniqueName: \"kubernetes.io/projected/4b2169c5-3da0-43b2-9e01-3bb431aa940e-kube-api-access-fr6p5\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.790387 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.790323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.790387 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.790379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd475\" (UniqueName: \"kubernetes.io/projected/82946fd3-0ef7-4f51-8760-85524d2f27e5-kube-api-access-sd475\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.792397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.792367    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82946fd3-0ef7-4f51-8760-85524d2f27e5-tmp\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.792638 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.792617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82946fd3-0ef7-4f51-8760-85524d2f27e5-tls-certs\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.798363 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.798344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd475\" (UniqueName: \"kubernetes.io/projected/82946fd3-0ef7-4f51-8760-85524d2f27e5-kube-api-access-sd475\") pod \"kube-auth-proxy-7f4c4dbc87-6x4kh\" (UID: \"82946fd3-0ef7-4f51-8760-85524d2f27e5\") " pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:43.891488 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.891396    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6p5\" (UniqueName: \"kubernetes.io/projected/4b2169c5-3da0-43b2-9e01-3bb431aa940e-kube-api-access-fr6p5\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.891488 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.891431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.891673 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:43.891601    2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 22 18:22:43.891673 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:43.891669    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert podName:4b2169c5-3da0-43b2-9e01-3bb431aa940e nodeName:}" failed. No retries permitted until 2026-07-22 18:22:44.391649472 +0000 UTC m=+518.607296853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert") pod "odh-model-controller-7998c67dc4-sfjkm" (UID: "4b2169c5-3da0-43b2-9e01-3bb431aa940e") : secret "odh-model-controller-webhook-cert" not found
Jul 22 18:22:43.901962 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.901936    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6p5\" (UniqueName: \"kubernetes.io/projected/4b2169c5-3da0-43b2-9e01-3bb431aa940e-kube-api-access-fr6p5\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:43.911707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:43.911683    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"
Jul 22 18:22:44.052336 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.052303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh"]
Jul 22 18:22:44.055306 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:44.055274    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82946fd3_0ef7_4f51_8760_85524d2f27e5.slice/crio-28fcd78dffba9ab51578c9d0449e53877b1db06c9db22191e55fffb7a39c3c8f WatchSource:0}: Error finding container 28fcd78dffba9ab51578c9d0449e53877b1db06c9db22191e55fffb7a39c3c8f: Status 404 returned error can't find the container with id 28fcd78dffba9ab51578c9d0449e53877b1db06c9db22191e55fffb7a39c3c8f
Jul 22 18:22:44.396033 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.395998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:44.398497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.398468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4b2169c5-3da0-43b2-9e01-3bb431aa940e-cert\") pod \"odh-model-controller-7998c67dc4-sfjkm\" (UID: \"4b2169c5-3da0-43b2-9e01-3bb431aa940e\") " pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:44.651750 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.651667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:44.697779 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.697743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m98r2"]
Jul 22 18:22:44.700216 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.700189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:44.703761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.703732    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 22 18:22:44.703884 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.703766    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-prphz\""
Jul 22 18:22:44.711691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.711624    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m98r2"]
Jul 22 18:22:44.790174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.790142    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-sfjkm"]
Jul 22 18:22:44.793306 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:44.793268    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2169c5_3da0_43b2_9e01_3bb431aa940e.slice/crio-2ea27452f9b29e5107a8b1ce73dd186af5c8db177f4ef4f4b8e4132a9cd8f537 WatchSource:0}: Error finding container 2ea27452f9b29e5107a8b1ce73dd186af5c8db177f4ef4f4b8e4132a9cd8f537: Status 404 returned error can't find the container with id 2ea27452f9b29e5107a8b1ce73dd186af5c8db177f4ef4f4b8e4132a9cd8f537
Jul 22 18:22:44.798736 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.798701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbpk\" (UniqueName: \"kubernetes.io/projected/c8e501ed-a65b-48eb-834d-8d961a16c0a0-kube-api-access-dtbpk\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:44.798840 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.798773    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:44.877982 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.877944    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:44.881281 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.881258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:44.883863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.883838    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 22 18:22:44.883863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.883851    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 22 18:22:44.884061 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.883947    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rq6tx\""
Jul 22 18:22:44.884121 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.884109    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 22 18:22:44.884182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.884164    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 22 18:22:44.899783 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.899744    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbpk\" (UniqueName: \"kubernetes.io/projected/c8e501ed-a65b-48eb-834d-8d961a16c0a0-kube-api-access-dtbpk\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:44.899930 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.899798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:44.899994 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:44.899929    2580 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 22 18:22:44.899994 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:44.899988    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs podName:c8e501ed-a65b-48eb-834d-8d961a16c0a0 nodeName:}" failed. No retries permitted until 2026-07-22 18:22:45.399969439 +0000 UTC m=+519.615616828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs") pod "model-serving-api-54d89bd5fc-m98r2" (UID: "c8e501ed-a65b-48eb-834d-8d961a16c0a0") : secret "model-serving-api-tls" not found
Jul 22 18:22:44.918989 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.918911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:44.925048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:44.925021    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbpk\" (UniqueName: \"kubernetes.io/projected/c8e501ed-a65b-48eb-834d-8d961a16c0a0-kube-api-access-dtbpk\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:45.000910 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.000877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.000928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.000950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.000983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.001008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.001050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.001113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.001089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bdmc\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.020890 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.020847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm" event={"ID":"4b2169c5-3da0-43b2-9e01-3bb431aa940e","Type":"ContainerStarted","Data":"2ea27452f9b29e5107a8b1ce73dd186af5c8db177f4ef4f4b8e4132a9cd8f537"}
Jul 22 18:22:45.022140 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.022116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh" event={"ID":"82946fd3-0ef7-4f51-8760-85524d2f27e5","Type":"ContainerStarted","Data":"28fcd78dffba9ab51578c9d0449e53877b1db06c9db22191e55fffb7a39c3c8f"}
Jul 22 18:22:45.102059 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bdmc\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102152    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102208    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102238    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.102497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.102281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.103071 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.103046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.105142 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.105117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.105249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.105232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.105573 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.105549    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.105654 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.105624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.111346 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.111324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.111478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.111455    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bdmc\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc\") pod \"istiod-openshift-gateway-7574459656-n72sq\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.192223 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.192189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:45.344016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.343978    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:45.347494 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:45.347458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e9bf67_e568_4e74_8bd3_dacce8733863.slice/crio-9a53ee5cf091b58e6c25b353c49e43afc99d6e7cae50b7d751857d52b969f06b WatchSource:0}: Error finding container 9a53ee5cf091b58e6c25b353c49e43afc99d6e7cae50b7d751857d52b969f06b: Status 404 returned error can't find the container with id 9a53ee5cf091b58e6c25b353c49e43afc99d6e7cae50b7d751857d52b969f06b
Jul 22 18:22:45.405068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.405030    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:45.407769 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.407748    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8e501ed-a65b-48eb-834d-8d961a16c0a0-tls-certs\") pod \"model-serving-api-54d89bd5fc-m98r2\" (UID: \"c8e501ed-a65b-48eb-834d-8d961a16c0a0\") " pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:45.613083 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.612990    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:45.800670 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:45.800527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-m98r2"]
Jul 22 18:22:45.804511 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:45.804482    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e501ed_a65b_48eb_834d_8d961a16c0a0.slice/crio-e2fa57bfba565b57f9adffb6c35565b5a0e29a437edbadde2b614dfb6e55bf8b WatchSource:0}: Error finding container e2fa57bfba565b57f9adffb6c35565b5a0e29a437edbadde2b614dfb6e55bf8b: Status 404 returned error can't find the container with id e2fa57bfba565b57f9adffb6c35565b5a0e29a437edbadde2b614dfb6e55bf8b
Jul 22 18:22:46.030241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:46.030162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2" event={"ID":"c8e501ed-a65b-48eb-834d-8d961a16c0a0","Type":"ContainerStarted","Data":"e2fa57bfba565b57f9adffb6c35565b5a0e29a437edbadde2b614dfb6e55bf8b"}
Jul 22 18:22:46.031940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:46.031907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" event={"ID":"46e9bf67-e568-4e74-8bd3-dacce8733863","Type":"ContainerStarted","Data":"9a53ee5cf091b58e6c25b353c49e43afc99d6e7cae50b7d751857d52b969f06b"}
Jul 22 18:22:49.054847 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:49.054714    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh" event={"ID":"82946fd3-0ef7-4f51-8760-85524d2f27e5","Type":"ContainerStarted","Data":"733916e42c7ee4b9d41024bc26a9f18befc324153469741a74ed34ba6eb670a4"}
Jul 22 18:22:49.056781 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:49.056749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm" event={"ID":"4b2169c5-3da0-43b2-9e01-3bb431aa940e","Type":"ContainerStarted","Data":"ad509c32740c726fdcbae5bd1502d64a91d57f1d8155c5f79bd9a09f0342a067"}
Jul 22 18:22:49.056953 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:49.056871    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:22:49.072425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:49.072357    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f4c4dbc87-6x4kh" podStartSLOduration=1.7703861779999999 podStartE2EDuration="6.072340998s" totalImagesPullingTime="4.30195482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:49.070757456 +0000 UTC m=+523.286404871" watchObservedRunningTime="2026-07-22 18:22:49.072340998 +0000 UTC m=+523.287988394"
Jul 22 18:22:49.086937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:49.086871    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm" podStartSLOduration=2.508792186 podStartE2EDuration="6.086847067s" totalImagesPullingTime="3.578054881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:49.085478892 +0000 UTC m=+523.301126297" watchObservedRunningTime="2026-07-22 18:22:49.086847067 +0000 UTC m=+523.302494471"
Jul 22 18:22:50.339688 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:50.339646    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 18:22:50.339998 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:50.339717    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 22 18:22:51.065990 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.065947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2" event={"ID":"c8e501ed-a65b-48eb-834d-8d961a16c0a0","Type":"ContainerStarted","Data":"366f6e6f2253e843ceb7e168a22cea3918157082b723c18f4a8e023ca611bda2"}
Jul 22 18:22:51.066206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.066052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:22:51.067523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.067496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" event={"ID":"46e9bf67-e568-4e74-8bd3-dacce8733863","Type":"ContainerStarted","Data":"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"}
Jul 22 18:22:51.067691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.067677    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:51.069369 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.069345    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-n72sq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 22 18:22:51.069463 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.069388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 22 18:22:51.082023 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.081973    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2" podStartSLOduration=2.552104451 podStartE2EDuration="7.081955675s" totalImagesPullingTime="4.529851224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:51.081334664 +0000 UTC m=+525.296982081" watchObservedRunningTime="2026-07-22 18:22:51.081955675 +0000 UTC m=+525.297603081"
Jul 22 18:22:51.100132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:51.100086    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" podStartSLOduration=2.110570634 podStartE2EDuration="7.100066644s" totalImagesPullingTime="4.98949601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:51.099613603 +0000 UTC m=+525.315261007" watchObservedRunningTime="2026-07-22 18:22:51.100066644 +0000 UTC m=+525.315714045"
Jul 22 18:22:52.072049 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:52.072024    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:55.055964 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.055928    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"]
Jul 22 18:22:55.058643 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.058622    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.061473 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.061435    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 22 18:22:55.062383 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.062362    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kww2t\""
Jul 22 18:22:55.063493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.063463    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"]
Jul 22 18:22:55.065730 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.065708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.068392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.068364    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-zghsf\""
Jul 22 18:22:55.068928 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.068908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 22 18:22:55.074868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.074781    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"]
Jul 22 18:22:55.077850 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.077830    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"]
Jul 22 18:22:55.197663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.197626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4bs\" (UniqueName: \"kubernetes.io/projected/ec746d59-da6e-44ab-bcb0-8c28d42de061-kube-api-access-wk4bs\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.197897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.197706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdfb\" (UniqueName: \"kubernetes.io/projected/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-kube-api-access-6sdfb\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.197897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.197742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.197897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.197774    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.298992 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.298953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.299285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.299026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4bs\" (UniqueName: \"kubernetes.io/projected/ec746d59-da6e-44ab-bcb0-8c28d42de061-kube-api-access-wk4bs\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.299285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.299077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdfb\" (UniqueName: \"kubernetes.io/projected/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-kube-api-access-6sdfb\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.299285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.299106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.299285 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:55.299216    2580 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 22 18:22:55.299285 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:55.299279    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert podName:ec746d59-da6e-44ab-bcb0-8c28d42de061 nodeName:}" failed. No retries permitted until 2026-07-22 18:22:55.799259554 +0000 UTC m=+530.014906953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert") pod "kserve-controller-manager-d6f4b65b8-5x9dd" (UID: "ec746d59-da6e-44ab-bcb0-8c28d42de061") : secret "kserve-webhook-server-cert" not found
Jul 22 18:22:55.301453 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.301432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.311593 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.311527    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4bs\" (UniqueName: \"kubernetes.io/projected/ec746d59-da6e-44ab-bcb0-8c28d42de061-kube-api-access-wk4bs\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.311718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.311700    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdfb\" (UniqueName: \"kubernetes.io/projected/e2dbfcfa-8938-4fc2-9727-38c8fca47faa-kube-api-access-6sdfb\") pod \"llmisvc-controller-manager-777c6bd7d5-7gq8j\" (UID: \"e2dbfcfa-8938-4fc2-9727-38c8fca47faa\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.379065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.379020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:55.508887 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.508856    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"]
Jul 22 18:22:55.511221 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:55.511187    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode2dbfcfa_8938_4fc2_9727_38c8fca47faa.slice/crio-2e981a480a128faff4bd7f2b88180d560d5b6db7b61c573bb3d71304da1a4f94 WatchSource:0}: Error finding container 2e981a480a128faff4bd7f2b88180d560d5b6db7b61c573bb3d71304da1a4f94: Status 404 returned error can't find the container with id 2e981a480a128faff4bd7f2b88180d560d5b6db7b61c573bb3d71304da1a4f94
Jul 22 18:22:55.801735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.801692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.804126 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.804104    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ec746d59-da6e-44ab-bcb0-8c28d42de061-cert\") pod \"kserve-controller-manager-d6f4b65b8-5x9dd\" (UID: \"ec746d59-da6e-44ab-bcb0-8c28d42de061\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:55.969537 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:55.969498    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:22:56.095906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:56.095824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j" event={"ID":"e2dbfcfa-8938-4fc2-9727-38c8fca47faa","Type":"ContainerStarted","Data":"2e981a480a128faff4bd7f2b88180d560d5b6db7b61c573bb3d71304da1a4f94"}
Jul 22 18:22:56.098612 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:56.098587    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"]
Jul 22 18:22:56.101463 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:22:56.101438    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec746d59_da6e_44ab_bcb0_8c28d42de061.slice/crio-d91880c5b31ad11be0deca9e095e12020d0e62015afd2f7835151b822a243c42 WatchSource:0}: Error finding container d91880c5b31ad11be0deca9e095e12020d0e62015afd2f7835151b822a243c42: Status 404 returned error can't find the container with id d91880c5b31ad11be0deca9e095e12020d0e62015afd2f7835151b822a243c42
Jul 22 18:22:57.102508 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:57.102467    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd" event={"ID":"ec746d59-da6e-44ab-bcb0-8c28d42de061","Type":"ContainerStarted","Data":"d91880c5b31ad11be0deca9e095e12020d0e62015afd2f7835151b822a243c42"}
Jul 22 18:22:58.667624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:58.667598    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:58.667961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:58.667757    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerName="discovery" containerID="cri-o://07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f" gracePeriod=30
Jul 22 18:22:58.915527 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:58.915504    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:59.031168 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031179    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031241    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031475 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031345    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031475 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031385    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031475 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031459    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.031638 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.031551    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bdmc\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc\") pod \"46e9bf67-e568-4e74-8bd3-dacce8733863\" (UID: \"46e9bf67-e568-4e74-8bd3-dacce8733863\") "
Jul 22 18:22:59.032983 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.032547    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:22:59.034353 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.034323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc" (OuterVolumeSpecName: "kube-api-access-7bdmc") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "kube-api-access-7bdmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:22:59.034468 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.034329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:22:59.034468 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.034435    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:22:59.034468 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.034462    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs" (OuterVolumeSpecName: "local-certs") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:22:59.035025 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.034995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token" (OuterVolumeSpecName: "istio-token") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:22:59.035689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.035663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts" (OuterVolumeSpecName: "cacerts") pod "46e9bf67-e568-4e74-8bd3-dacce8733863" (UID: "46e9bf67-e568-4e74-8bd3-dacce8733863"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:22:59.112906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.112863    2580 generic.go:356] "Generic (PLEG): container finished" podID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerID="07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f" exitCode=0
Jul 22 18:22:59.113093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.112925    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"
Jul 22 18:22:59.113093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.112946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" event={"ID":"46e9bf67-e568-4e74-8bd3-dacce8733863","Type":"ContainerDied","Data":"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"}
Jul 22 18:22:59.113093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.113030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-n72sq" event={"ID":"46e9bf67-e568-4e74-8bd3-dacce8733863","Type":"ContainerDied","Data":"9a53ee5cf091b58e6c25b353c49e43afc99d6e7cae50b7d751857d52b969f06b"}
Jul 22 18:22:59.113093 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.113059    2580 scope.go:117] "RemoveContainer" containerID="07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"
Jul 22 18:22:59.114539 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.114505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j" event={"ID":"e2dbfcfa-8938-4fc2-9727-38c8fca47faa","Type":"ContainerStarted","Data":"11250adc646a3c6773dedab33c0730d149ff620376dec56336d6e8dd3a747ede"}
Jul 22 18:22:59.114662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.114643    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:22:59.121911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.121895    2580 scope.go:117] "RemoveContainer" containerID="07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"
Jul 22 18:22:59.122194 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:22:59.122168    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f\": container with ID starting with 07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f not found: ID does not exist" containerID="07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"
Jul 22 18:22:59.122239 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.122205    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f"} err="failed to get container status \"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f\": rpc error: code = NotFound desc = could not find container \"07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f\": container with ID starting with 07cf7cb57bd38bf8bae8e4bee438056eb68aae150dfb37197c8486ad4ef1024f not found: ID does not exist"
Jul 22 18:22:59.132261 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132227    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-dns-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132261 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132251    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-token\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132265    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-kubeconfig\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132277    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/46e9bf67-e568-4e74-8bd3-dacce8733863-istio-csr-ca-configmap\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132289    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bdmc\" (UniqueName: \"kubernetes.io/projected/46e9bf67-e568-4e74-8bd3-dacce8733863-kube-api-access-7bdmc\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132301    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/46e9bf67-e568-4e74-8bd3-dacce8733863-cacerts\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.132397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.132314    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e9bf67-e568-4e74-8bd3-dacce8733863-local-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:22:59.145634 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.145568    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:59.150252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.150230    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-n72sq"]
Jul 22 18:22:59.183388 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:22:59.183324    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j" podStartSLOduration=1.053179388 podStartE2EDuration="4.18330388s" totalImagesPullingTime="3.130124492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:22:59.181255768 +0000 UTC m=+533.396903182" watchObservedRunningTime="2026-07-22 18:22:59.18330388 +0000 UTC m=+533.398951284"
Jul 22 18:23:00.063456 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:00.063419    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-sfjkm"
Jul 22 18:23:00.320131 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:00.320049    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" path="/var/lib/kubelet/pods/46e9bf67-e568-4e74-8bd3-dacce8733863/volumes"
Jul 22 18:23:01.126233 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:01.126156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd" event={"ID":"ec746d59-da6e-44ab-bcb0-8c28d42de061","Type":"ContainerStarted","Data":"ad6c5d16ebdaaf349ee63f9ffe79775df309f4dc5301c96d31ee7de645f1d915"}
Jul 22 18:23:01.126594 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:01.126295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:23:01.141666 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:01.141591    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd" podStartSLOduration=1.473897263 podStartE2EDuration="6.141569406s" totalImagesPullingTime="4.667672143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:23:01.141502894 +0000 UTC m=+535.357150296" watchObservedRunningTime="2026-07-22 18:23:01.141569406 +0000 UTC m=+535.357216809"
Jul 22 18:23:02.075410 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:02.075377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-m98r2"
Jul 22 18:23:30.123425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:30.123393    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7gq8j"
Jul 22 18:23:32.135941 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:32.135912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5x9dd"
Jul 22 18:23:44.446639 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.446607    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:44.447112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.446927    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerName="discovery"
Jul 22 18:23:44.447112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.446938    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerName="discovery"
Jul 22 18:23:44.447112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.446985    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="46e9bf67-e568-4e74-8bd3-dacce8733863" containerName="discovery"
Jul 22 18:23:44.448870 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.448853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:44.451319 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.451295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-ljs7r\""
Jul 22 18:23:44.451685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.451668    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 22 18:23:44.455928 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.455910    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 22 18:23:44.472340 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.472307    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:44.585355 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.585324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fbt\" (UniqueName: \"kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt\") pod \"kuadrant-operator-catalog-lv4ns\" (UID: \"e9be43ae-c071-4a83-85ed-3057e8a39515\") " pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:44.686210 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.686171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fbt\" (UniqueName: \"kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt\") pod \"kuadrant-operator-catalog-lv4ns\" (UID: \"e9be43ae-c071-4a83-85ed-3057e8a39515\") " pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:44.694512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.694477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fbt\" (UniqueName: \"kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt\") pod \"kuadrant-operator-catalog-lv4ns\" (UID: \"e9be43ae-c071-4a83-85ed-3057e8a39515\") " pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:44.757549 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.757457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:44.812776 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.812743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:44.884150 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:44.884121    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:44.887434 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:23:44.887406    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9be43ae_c071_4a83_85ed_3057e8a39515.slice/crio-70a3604dc22ba5ce810d477926bf472b34a519fab785a4cec0a4e170fb6df229 WatchSource:0}: Error finding container 70a3604dc22ba5ce810d477926bf472b34a519fab785a4cec0a4e170fb6df229: Status 404 returned error can't find the container with id 70a3604dc22ba5ce810d477926bf472b34a519fab785a4cec0a4e170fb6df229
Jul 22 18:23:45.022356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.022274    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jn9p8"]
Jul 22 18:23:45.025665 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.025648    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:45.032587 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.032544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jn9p8"]
Jul 22 18:23:45.089336 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.089298    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbrh\" (UniqueName: \"kubernetes.io/projected/fa89d916-c291-4fc3-87d7-605d18ae5cc9-kube-api-access-5zbrh\") pod \"kuadrant-operator-catalog-jn9p8\" (UID: \"fa89d916-c291-4fc3-87d7-605d18ae5cc9\") " pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:45.190126 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.190086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbrh\" (UniqueName: \"kubernetes.io/projected/fa89d916-c291-4fc3-87d7-605d18ae5cc9-kube-api-access-5zbrh\") pod \"kuadrant-operator-catalog-jn9p8\" (UID: \"fa89d916-c291-4fc3-87d7-605d18ae5cc9\") " pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:45.198548 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.198519    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbrh\" (UniqueName: \"kubernetes.io/projected/fa89d916-c291-4fc3-87d7-605d18ae5cc9-kube-api-access-5zbrh\") pod \"kuadrant-operator-catalog-jn9p8\" (UID: \"fa89d916-c291-4fc3-87d7-605d18ae5cc9\") " pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:45.302926 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.302840    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" event={"ID":"e9be43ae-c071-4a83-85ed-3057e8a39515","Type":"ContainerStarted","Data":"70a3604dc22ba5ce810d477926bf472b34a519fab785a4cec0a4e170fb6df229"}
Jul 22 18:23:45.335732 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.335697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:45.464559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:45.464527    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jn9p8"]
Jul 22 18:23:45.467614 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:23:45.467586    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa89d916_c291_4fc3_87d7_605d18ae5cc9.slice/crio-6100e9155171d3fca7105274e0fdf2924ee3f662d1b1f00782366924947f612c WatchSource:0}: Error finding container 6100e9155171d3fca7105274e0fdf2924ee3f662d1b1f00782366924947f612c: Status 404 returned error can't find the container with id 6100e9155171d3fca7105274e0fdf2924ee3f662d1b1f00782366924947f612c
Jul 22 18:23:46.310726 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:46.310649    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8" event={"ID":"fa89d916-c291-4fc3-87d7-605d18ae5cc9","Type":"ContainerStarted","Data":"6100e9155171d3fca7105274e0fdf2924ee3f662d1b1f00782366924947f612c"}
Jul 22 18:23:48.325677 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.325627    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" podUID="e9be43ae-c071-4a83-85ed-3057e8a39515" containerName="registry-server" containerID="cri-o://7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3" gracePeriod=2
Jul 22 18:23:48.325677 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.325573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" event={"ID":"e9be43ae-c071-4a83-85ed-3057e8a39515","Type":"ContainerStarted","Data":"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"}
Jul 22 18:23:48.327083 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.327063    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8" event={"ID":"fa89d916-c291-4fc3-87d7-605d18ae5cc9","Type":"ContainerStarted","Data":"9897fac7c7c43b0fea42fceda3095805769afcda24d4ba434f690a3a5e830543"}
Jul 22 18:23:48.340334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.340288    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" podStartSLOduration=1.9385101219999998 podStartE2EDuration="4.340273844s" totalImagesPullingTime="2.401763722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:23:48.339364136 +0000 UTC m=+582.555011540" watchObservedRunningTime="2026-07-22 18:23:48.340273844 +0000 UTC m=+582.555921246"
Jul 22 18:23:48.353273 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.353231    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8" podStartSLOduration=1.518713124 podStartE2EDuration="3.35321369s" totalImagesPullingTime="1.834500566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:23:48.352470654 +0000 UTC m=+582.568118058" watchObservedRunningTime="2026-07-22 18:23:48.35321369 +0000 UTC m=+582.568861083"
Jul 22 18:23:48.559625 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.559600    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:48.620632 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.620534    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4fbt\" (UniqueName: \"kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt\") pod \"e9be43ae-c071-4a83-85ed-3057e8a39515\" (UID: \"e9be43ae-c071-4a83-85ed-3057e8a39515\") "
Jul 22 18:23:48.622682 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.622646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt" (OuterVolumeSpecName: "kube-api-access-z4fbt") pod "e9be43ae-c071-4a83-85ed-3057e8a39515" (UID: "e9be43ae-c071-4a83-85ed-3057e8a39515"). InnerVolumeSpecName "kube-api-access-z4fbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:23:48.721217 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:48.721169    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4fbt\" (UniqueName: \"kubernetes.io/projected/e9be43ae-c071-4a83-85ed-3057e8a39515-kube-api-access-z4fbt\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:23:49.332555 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.332520    2580 generic.go:356] "Generic (PLEG): container finished" podID="e9be43ae-c071-4a83-85ed-3057e8a39515" containerID="7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3" exitCode=0
Jul 22 18:23:49.333002 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.332564    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns"
Jul 22 18:23:49.333002 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.332558    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" event={"ID":"e9be43ae-c071-4a83-85ed-3057e8a39515","Type":"ContainerDied","Data":"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"}
Jul 22 18:23:49.333002 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.332682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lv4ns" event={"ID":"e9be43ae-c071-4a83-85ed-3057e8a39515","Type":"ContainerDied","Data":"70a3604dc22ba5ce810d477926bf472b34a519fab785a4cec0a4e170fb6df229"}
Jul 22 18:23:49.333002 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.332702    2580 scope.go:117] "RemoveContainer" containerID="7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"
Jul 22 18:23:49.341781 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.341764    2580 scope.go:117] "RemoveContainer" containerID="7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"
Jul 22 18:23:49.342057 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:23:49.342034    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3\": container with ID starting with 7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3 not found: ID does not exist" containerID="7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"
Jul 22 18:23:49.342108 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.342065    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3"} err="failed to get container status \"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3\": rpc error: code = NotFound desc = could not find container \"7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3\": container with ID starting with 7682d78c5863626e586642b71a53305a2729c9dc4d26b9ee1789932a92074dc3 not found: ID does not exist"
Jul 22 18:23:49.352435 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.352414    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:49.355957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:49.355935    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lv4ns"]
Jul 22 18:23:50.319908 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:50.319877    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9be43ae-c071-4a83-85ed-3057e8a39515" path="/var/lib/kubelet/pods/e9be43ae-c071-4a83-85ed-3057e8a39515/volumes"
Jul 22 18:23:55.335989 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:55.335945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:55.336478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:55.336051    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:55.358912 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:55.358886    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:23:55.381226 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:23:55.381201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jn9p8"
Jul 22 18:24:06.966147 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.966108    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f79954d97-dnh22"]
Jul 22 18:24:06.966632 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.966532    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9be43ae-c071-4a83-85ed-3057e8a39515" containerName="registry-server"
Jul 22 18:24:06.966632 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.966580    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9be43ae-c071-4a83-85ed-3057e8a39515" containerName="registry-server"
Jul 22 18:24:06.966744 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.966694    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9be43ae-c071-4a83-85ed-3057e8a39515" containerName="registry-server"
Jul 22 18:24:06.969589 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.969569    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:06.984409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:06.984382    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f79954d97-dnh22"]
Jul 22 18:24:07.075560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075522    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56wwm\" (UniqueName: \"kubernetes.io/projected/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-kube-api-access-56wwm\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-trusted-ca-bundle\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-service-ca\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-oauth-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.075971 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.075863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-oauth-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.176758 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176719    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.176758 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-service-ca\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-oauth-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176837    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-oauth-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56wwm\" (UniqueName: \"kubernetes.io/projected/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-kube-api-access-56wwm\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.176962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-trusted-ca-bundle\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177592 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.177560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-service-ca\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177706 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.177660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-oauth-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177754 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.177720    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.177840 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.177801    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-trusted-ca-bundle\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.179773 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.179747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-oauth-config\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.179979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.179959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-console-serving-cert\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.185334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.185310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56wwm\" (UniqueName: \"kubernetes.io/projected/51a7d91a-dcd8-4c1c-859a-c022a37c3b2e-kube-api-access-56wwm\") pod \"console-f79954d97-dnh22\" (UID: \"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e\") " pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.279050 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.278964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:07.409046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:07.409006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f79954d97-dnh22"]
Jul 22 18:24:07.411425 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:07.411396    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a7d91a_dcd8_4c1c_859a_c022a37c3b2e.slice/crio-288b415773a7e5103b48f09c43748d60cbd1e43b540a494fd56ba46f50b2258c WatchSource:0}: Error finding container 288b415773a7e5103b48f09c43748d60cbd1e43b540a494fd56ba46f50b2258c: Status 404 returned error can't find the container with id 288b415773a7e5103b48f09c43748d60cbd1e43b540a494fd56ba46f50b2258c
Jul 22 18:24:08.412685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:08.412575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f79954d97-dnh22" event={"ID":"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e","Type":"ContainerStarted","Data":"f3b2286e6e46c0d549c063a8659954844fed97641d18258c41aa1bbc48788fc7"}
Jul 22 18:24:08.412685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:08.412621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f79954d97-dnh22" event={"ID":"51a7d91a-dcd8-4c1c-859a-c022a37c3b2e","Type":"ContainerStarted","Data":"288b415773a7e5103b48f09c43748d60cbd1e43b540a494fd56ba46f50b2258c"}
Jul 22 18:24:08.432895 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:08.432844    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f79954d97-dnh22" podStartSLOduration=2.432826383 podStartE2EDuration="2.432826383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:08.431674158 +0000 UTC m=+602.647321561" watchObservedRunningTime="2026-07-22 18:24:08.432826383 +0000 UTC m=+602.648473777"
Jul 22 18:24:17.279942 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.279909    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:17.280331 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.279984    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:17.284559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.284536    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:17.452675 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.452649    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f79954d97-dnh22"
Jul 22 18:24:17.497427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.497398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:24:17.788505 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.788468    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"]
Jul 22 18:24:17.791612 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.791590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.794505 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.794486    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-npxbv\""
Jul 22 18:24:17.808159 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.808138    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"]
Jul 22 18:24:17.866467 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.866437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.866592 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.866476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcmd\" (UniqueName: \"kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.967301 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.967268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcmd\" (UniqueName: \"kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.967448 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.967361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.967698 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.967682    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:17.979379 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:17.979348    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcmd\" (UniqueName: \"kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:18.102603 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:18.102514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:18.232476 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:18.232447    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"]
Jul 22 18:24:18.236106 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:18.236083    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b588ec5_62c7_4ab2_9bdc_68f256f65fea.slice/crio-936e6368ad43b646d5dcab83cfe37716222a813f3668bf024306d845ee3ed2df WatchSource:0}: Error finding container 936e6368ad43b646d5dcab83cfe37716222a813f3668bf024306d845ee3ed2df: Status 404 returned error can't find the container with id 936e6368ad43b646d5dcab83cfe37716222a813f3668bf024306d845ee3ed2df
Jul 22 18:24:18.453898 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:18.453868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs" event={"ID":"2b588ec5-62c7-4ab2-9bdc-68f256f65fea","Type":"ContainerStarted","Data":"936e6368ad43b646d5dcab83cfe37716222a813f3668bf024306d845ee3ed2df"}
Jul 22 18:24:20.419646 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.419613    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"]
Jul 22 18:24:20.421997 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.421979    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:20.425636 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.425620    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rvb55\""
Jul 22 18:24:20.437523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.437496    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"]
Jul 22 18:24:20.489060 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.489025    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvwk\" (UniqueName: \"kubernetes.io/projected/ee03b86e-1f4e-49f1-8f3c-42168ebfaab3-kube-api-access-jsvwk\") pod \"authorino-operator-75cc8d5cb7-nrs7h\" (UID: \"ee03b86e-1f4e-49f1-8f3c-42168ebfaab3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:20.590250 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.590216    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvwk\" (UniqueName: \"kubernetes.io/projected/ee03b86e-1f4e-49f1-8f3c-42168ebfaab3-kube-api-access-jsvwk\") pod \"authorino-operator-75cc8d5cb7-nrs7h\" (UID: \"ee03b86e-1f4e-49f1-8f3c-42168ebfaab3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:20.599698 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.599662    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvwk\" (UniqueName: \"kubernetes.io/projected/ee03b86e-1f4e-49f1-8f3c-42168ebfaab3-kube-api-access-jsvwk\") pod \"authorino-operator-75cc8d5cb7-nrs7h\" (UID: \"ee03b86e-1f4e-49f1-8f3c-42168ebfaab3\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:20.733011 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.732964    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:20.885579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:20.885542    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"]
Jul 22 18:24:20.893109 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:20.893072    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee03b86e_1f4e_49f1_8f3c_42168ebfaab3.slice/crio-958452017e7a163e94001df4600432576aa29ed0cc9bb1267876c8657d926c12 WatchSource:0}: Error finding container 958452017e7a163e94001df4600432576aa29ed0cc9bb1267876c8657d926c12: Status 404 returned error can't find the container with id 958452017e7a163e94001df4600432576aa29ed0cc9bb1267876c8657d926c12
Jul 22 18:24:21.469132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:21.469092    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h" event={"ID":"ee03b86e-1f4e-49f1-8f3c-42168ebfaab3","Type":"ContainerStarted","Data":"958452017e7a163e94001df4600432576aa29ed0cc9bb1267876c8657d926c12"}
Jul 22 18:24:22.656372 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.656335    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"]
Jul 22 18:24:22.662995 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.662973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:22.665630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.665603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 22 18:24:22.665760 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.665739    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wsccs\""
Jul 22 18:24:22.669393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.669370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"]
Jul 22 18:24:22.709932 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.709902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/919b3a0a-2915-4f52-8cf6-cae9adfa287a-kube-api-access-r98n2\") pod \"dns-operator-controller-manager-864dd78c4c-wjh69\" (UID: \"919b3a0a-2915-4f52-8cf6-cae9adfa287a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:22.810628 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.810589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/919b3a0a-2915-4f52-8cf6-cae9adfa287a-kube-api-access-r98n2\") pod \"dns-operator-controller-manager-864dd78c4c-wjh69\" (UID: \"919b3a0a-2915-4f52-8cf6-cae9adfa287a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:22.840784 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.840761    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98n2\" (UniqueName: \"kubernetes.io/projected/919b3a0a-2915-4f52-8cf6-cae9adfa287a-kube-api-access-r98n2\") pod \"dns-operator-controller-manager-864dd78c4c-wjh69\" (UID: \"919b3a0a-2915-4f52-8cf6-cae9adfa287a\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:22.977379 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:22.977344    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:24.688293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.688266    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"]
Jul 22 18:24:24.692909 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.692889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:24.695911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.695735    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-n9dm2\""
Jul 22 18:24:24.701374 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.701326    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"]
Jul 22 18:24:24.728171 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.728145    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgg8\" (UniqueName: \"kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8\") pod \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" (UID: \"1847b302-1b53-4f3d-a50e-521e22bdd784\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:24.760914 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.760899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"]
Jul 22 18:24:24.764559 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:24.764529    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919b3a0a_2915_4f52_8cf6_cae9adfa287a.slice/crio-21a2973da059e36bad2366f0e0af2ede25c3650284f07750be2b26f41038e98e WatchSource:0}: Error finding container 21a2973da059e36bad2366f0e0af2ede25c3650284f07750be2b26f41038e98e: Status 404 returned error can't find the container with id 21a2973da059e36bad2366f0e0af2ede25c3650284f07750be2b26f41038e98e
Jul 22 18:24:24.829517 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.829492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgg8\" (UniqueName: \"kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8\") pod \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" (UID: \"1847b302-1b53-4f3d-a50e-521e22bdd784\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:24.837911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:24.837889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgg8\" (UniqueName: \"kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8\") pod \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" (UID: \"1847b302-1b53-4f3d-a50e-521e22bdd784\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:25.008535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.008505    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:25.148508 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.148483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"]
Jul 22 18:24:25.149926 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:25.149899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1847b302_1b53_4f3d_a50e_521e22bdd784.slice/crio-71070ed4ad3f8ea7bd439eba3f459f78f77b726fe1f1bb29f7f8501a128731c5 WatchSource:0}: Error finding container 71070ed4ad3f8ea7bd439eba3f459f78f77b726fe1f1bb29f7f8501a128731c5: Status 404 returned error can't find the container with id 71070ed4ad3f8ea7bd439eba3f459f78f77b726fe1f1bb29f7f8501a128731c5
Jul 22 18:24:25.491617 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.491579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs" event={"ID":"2b588ec5-62c7-4ab2-9bdc-68f256f65fea","Type":"ContainerStarted","Data":"e6925e469cd66352c3d26fd6a1dec198439033a7994ce5355dcc7efaeb441909"}
Jul 22 18:24:25.491841 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.491775    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:25.493262 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.493243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h" event={"ID":"ee03b86e-1f4e-49f1-8f3c-42168ebfaab3","Type":"ContainerStarted","Data":"f74dfce226cfba6825e883d4530f31b738094e792c4a674ab47cd5c5e9e8685c"}
Jul 22 18:24:25.493353 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.493329    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:25.494480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.494457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69" event={"ID":"919b3a0a-2915-4f52-8cf6-cae9adfa287a","Type":"ContainerStarted","Data":"21a2973da059e36bad2366f0e0af2ede25c3650284f07750be2b26f41038e98e"}
Jul 22 18:24:25.495689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.495672    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw" event={"ID":"1847b302-1b53-4f3d-a50e-521e22bdd784","Type":"ContainerStarted","Data":"71070ed4ad3f8ea7bd439eba3f459f78f77b726fe1f1bb29f7f8501a128731c5"}
Jul 22 18:24:25.511206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.511167    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs" podStartSLOduration=2.117133849 podStartE2EDuration="8.51115601s" totalImagesPullingTime="6.394022161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:25.510268179 +0000 UTC m=+619.725915580" watchObservedRunningTime="2026-07-22 18:24:25.51115601 +0000 UTC m=+619.726803411"
Jul 22 18:24:25.531224 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:25.531179    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h" podStartSLOduration=1.8061778529999999 podStartE2EDuration="5.531165264s" totalImagesPullingTime="3.724987411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:25.529898503 +0000 UTC m=+619.745545898" watchObservedRunningTime="2026-07-22 18:24:25.531165264 +0000 UTC m=+619.746812667"
Jul 22 18:24:28.059979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.059947    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"]
Jul 22 18:24:28.062414 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.062395    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.065025 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.064965    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 22 18:24:28.065148 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.065048    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7qf7d\""
Jul 22 18:24:28.065148 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.065051    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 22 18:24:28.071676 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.071654    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"]
Jul 22 18:24:28.160772 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.160740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.160961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.160777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjxx\" (UniqueName: \"kubernetes.io/projected/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-kube-api-access-fdjxx\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.160961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.160869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.261654 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.261620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.261863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.261662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjxx\" (UniqueName: \"kubernetes.io/projected/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-kube-api-access-fdjxx\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.261863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.261697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.261863 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:28.261759    2580 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 22 18:24:28.261863 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:28.261847    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert podName:3f6a68df-04f0-4223-a91d-eca23e6ea3a0 nodeName:}" failed. No retries permitted until 2026-07-22 18:24:28.761829698 +0000 UTC m=+622.977477094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-85h4m" (UID: "3f6a68df-04f0-4223-a91d-eca23e6ea3a0") : secret "plugin-serving-cert" not found
Jul 22 18:24:28.262287 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.262270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.272463 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.272437    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjxx\" (UniqueName: \"kubernetes.io/projected/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-kube-api-access-fdjxx\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.510254 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.510218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69" event={"ID":"919b3a0a-2915-4f52-8cf6-cae9adfa287a","Type":"ContainerStarted","Data":"1c36fa9ac19fbc79dfc44e3832026cc992ae2703270198aab9dad20ac146ea53"}
Jul 22 18:24:28.510431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.510313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:28.511715 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.511691    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw" event={"ID":"1847b302-1b53-4f3d-a50e-521e22bdd784","Type":"ContainerStarted","Data":"ceed81b1d203a18bc1321f0597dc1109d4d3b38c0274a2dc7f280a9dd259405d"}
Jul 22 18:24:28.511827 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.511731    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:28.526395 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.526353    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69" podStartSLOduration=3.667322144 podStartE2EDuration="6.526339473s" totalImagesPullingTime="2.859017329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:28.525663776 +0000 UTC m=+622.741311179" watchObservedRunningTime="2026-07-22 18:24:28.526339473 +0000 UTC m=+622.741986876"
Jul 22 18:24:28.542337 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.542302    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw" podStartSLOduration=2.066772317 podStartE2EDuration="4.54229094s" totalImagesPullingTime="2.475518623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:28.541461308 +0000 UTC m=+622.757108712" watchObservedRunningTime="2026-07-22 18:24:28.54229094 +0000 UTC m=+622.757938342"
Jul 22 18:24:28.766035 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.765939    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.768375 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.768351    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f6a68df-04f0-4223-a91d-eca23e6ea3a0-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-85h4m\" (UID: \"3f6a68df-04f0-4223-a91d-eca23e6ea3a0\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:28.971920 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:28.971887    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"
Jul 22 18:24:29.097426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:29.097396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m"]
Jul 22 18:24:29.100571 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:29.100547    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f6a68df_04f0_4223_a91d_eca23e6ea3a0.slice/crio-f3dbe010d593cffbe34e94f1abb9bd958e37396be61b783959329ac96f3b9334 WatchSource:0}: Error finding container f3dbe010d593cffbe34e94f1abb9bd958e37396be61b783959329ac96f3b9334: Status 404 returned error can't find the container with id f3dbe010d593cffbe34e94f1abb9bd958e37396be61b783959329ac96f3b9334
Jul 22 18:24:29.517563 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:29.517530    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m" event={"ID":"3f6a68df-04f0-4223-a91d-eca23e6ea3a0","Type":"ContainerStarted","Data":"f3dbe010d593cffbe34e94f1abb9bd958e37396be61b783959329ac96f3b9334"}
Jul 22 18:24:36.502358 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:36.502319    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:36.502948 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:36.502484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-nrs7h"
Jul 22 18:24:38.322854 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.322794    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"]
Jul 22 18:24:38.323347 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.322889    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"]
Jul 22 18:24:38.323347 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.323056    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" containerName="manager" containerID="cri-o://e6925e469cd66352c3d26fd6a1dec198439033a7994ce5355dcc7efaeb441909" gracePeriod=2
Jul 22 18:24:38.325236 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.325200    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:38.339517 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.339477    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:38.340010 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.339991    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" containerName="manager"
Jul 22 18:24:38.340010 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.340011    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" containerName="manager"
Jul 22 18:24:38.340143 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.340093    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" containerName="manager"
Jul 22 18:24:38.342495 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.342476    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.356893 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.356864    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:38.361246 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.361212    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"]
Jul 22 18:24:38.361512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.361482    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" containerName="manager" containerID="cri-o://ceed81b1d203a18bc1321f0597dc1109d4d3b38c0274a2dc7f280a9dd259405d" gracePeriod=2
Jul 22 18:24:38.363087 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.363068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:38.371097 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.371074    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"]
Jul 22 18:24:38.383413 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.383381    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:38.386879 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.386821    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:38.388836 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.388800    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"]
Jul 22 18:24:38.389385 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.389365    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" containerName="manager"
Jul 22 18:24:38.389482 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.389387    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" containerName="manager"
Jul 22 18:24:38.389482 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.389478    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" containerName="manager"
Jul 22 18:24:38.389606 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.389574    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:38.391961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.391935    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:38.403600 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.403577    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"]
Jul 22 18:24:38.417922 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.417886    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:38.419883 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:38.419850    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:38.448529 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.448476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.448629 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.448590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.448722 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.448700    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/70e234f8-4190-4869-917a-8d185c61380a-kube-api-access-bvr4c\") pod \"limitador-operator-controller-manager-859d7b78bd-sxgdc\" (UID: \"70e234f8-4190-4869-917a-8d185c61380a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:38.549959 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.549924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.550133 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.549988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.550133 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.550055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/70e234f8-4190-4869-917a-8d185c61380a-kube-api-access-bvr4c\") pod \"limitador-operator-controller-manager-859d7b78bd-sxgdc\" (UID: \"70e234f8-4190-4869-917a-8d185c61380a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:38.550446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.550417    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.559999 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.559975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ssxlp\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.560206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.560183    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvr4c\" (UniqueName: \"kubernetes.io/projected/70e234f8-4190-4869-917a-8d185c61380a-kube-api-access-bvr4c\") pod \"limitador-operator-controller-manager-859d7b78bd-sxgdc\" (UID: \"70e234f8-4190-4869-917a-8d185c61380a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:38.745252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.745219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:38.752091 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:38.752062    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:39.520315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:39.520284    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wjh69"
Jul 22 18:24:39.538558 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:39.538521    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:39.540304 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:39.540272    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:42.517667 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:42.517599    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74f47d4cdf-9zbpb" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerName="console" containerID="cri-o://d5aaf69491833e9123ada0faa4724f10beccafc30e3224fc9424a62d915604b2" gracePeriod=15
Jul 22 18:24:42.565915 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:42.565878    2580 patch_prober.go:28] interesting pod/console-74f47d4cdf-9zbpb container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused" start-of-body=
Jul 22 18:24:42.566095 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:42.565934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-74f47d4cdf-9zbpb" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused"
Jul 22 18:24:51.647487 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.647314    2580 generic.go:356] "Generic (PLEG): container finished" podID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" containerID="e6925e469cd66352c3d26fd6a1dec198439033a7994ce5355dcc7efaeb441909" exitCode=0
Jul 22 18:24:51.651173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.651004    2580 generic.go:356] "Generic (PLEG): container finished" podID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerID="d5aaf69491833e9123ada0faa4724f10beccafc30e3224fc9424a62d915604b2" exitCode=0
Jul 22 18:24:51.651173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.651096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f47d4cdf-9zbpb" event={"ID":"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd","Type":"ContainerDied","Data":"d5aaf69491833e9123ada0faa4724f10beccafc30e3224fc9424a62d915604b2"}
Jul 22 18:24:51.654650 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.654544    2580 generic.go:356] "Generic (PLEG): container finished" podID="1847b302-1b53-4f3d-a50e-521e22bdd784" containerID="ceed81b1d203a18bc1321f0597dc1109d4d3b38c0274a2dc7f280a9dd259405d" exitCode=0
Jul 22 18:24:51.754793 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.754268    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:51.758285 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:51.758241    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cb4c6e_f811_4aae_a78c_9f2b982a8e0b.slice/crio-fa729286c5c769dca94e2990b49c3a3710bbb5127cb61f67bc6c703cc9a26f40 WatchSource:0}: Error finding container fa729286c5c769dca94e2990b49c3a3710bbb5127cb61f67bc6c703cc9a26f40: Status 404 returned error can't find the container with id fa729286c5c769dca94e2990b49c3a3710bbb5127cb61f67bc6c703cc9a26f40
Jul 22 18:24:51.771779 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.771395    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:51.773516 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:51.773482    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:51.774972 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.774955    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:51.777287 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:51.777262    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:51.778954 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:51.778923    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:51.783098 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.783075    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"]
Jul 22 18:24:51.786676 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.786633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:24:51.787390 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:24:51.787368    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e234f8_4190_4869_917a_8d185c61380a.slice/crio-9e5038a7c7ad1bf575a8941c7dd0c13e5029a638123b839a2b0f6ec0b2abf43f WatchSource:0}: Error finding container 9e5038a7c7ad1bf575a8941c7dd0c13e5029a638123b839a2b0f6ec0b2abf43f: Status 404 returned error can't find the container with id 9e5038a7c7ad1bf575a8941c7dd0c13e5029a638123b839a2b0f6ec0b2abf43f
Jul 22 18:24:51.806338 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:51.806312    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:51.808063 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:51.808031    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:51.883589 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883550    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dcmd\" (UniqueName: \"kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd\") pod \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") "
Jul 22 18:24:51.883589 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.883868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.883868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22fj8\" (UniqueName: \"kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.883868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883661    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.883868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.883868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883800    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkgg8\" (UniqueName: \"kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8\") pod \"1847b302-1b53-4f3d-a50e-521e22bdd784\" (UID: \"1847b302-1b53-4f3d-a50e-521e22bdd784\") "
Jul 22 18:24:51.884102 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.884102 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config\") pod \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\" (UID: \"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd\") "
Jul 22 18:24:51.884102 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.883999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config" (OuterVolumeSpecName: "console-config") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:24:51.884102 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884033    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume\") pod \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\" (UID: \"2b588ec5-62c7-4ab2-9bdc-68f256f65fea\") "
Jul 22 18:24:51.884282 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884176    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:24:51.884282 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884195    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:24:51.884393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884331    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.884393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884348    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-oauth-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.884393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884361    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-trusted-ca-bundle\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.884567 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:24:51.884742 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.884712    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2b588ec5-62c7-4ab2-9bdc-68f256f65fea" (UID: "2b588ec5-62c7-4ab2-9bdc-68f256f65fea"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:24:51.886055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.886023    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd" (OuterVolumeSpecName: "kube-api-access-6dcmd") pod "2b588ec5-62c7-4ab2-9bdc-68f256f65fea" (UID: "2b588ec5-62c7-4ab2-9bdc-68f256f65fea"). InnerVolumeSpecName "kube-api-access-6dcmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:24:51.886160 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.886121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8" (OuterVolumeSpecName: "kube-api-access-22fj8") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "kube-api-access-22fj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:24:51.886160 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.886137    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:24:51.886547 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.886526    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" (UID: "6cab0f3f-5465-4c16-9193-e0e5acc1b2dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:24:51.886598 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.886568    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8" (OuterVolumeSpecName: "kube-api-access-hkgg8") pod "1847b302-1b53-4f3d-a50e-521e22bdd784" (UID: "1847b302-1b53-4f3d-a50e-521e22bdd784"). InnerVolumeSpecName "kube-api-access-hkgg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:24:51.984823 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984762    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcmd\" (UniqueName: \"kubernetes.io/projected/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-kube-api-access-6dcmd\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.984823 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984796    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.984823 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fj8\" (UniqueName: \"kubernetes.io/projected/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-kube-api-access-22fj8\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.985054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984840    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-serving-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.985054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984849    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkgg8\" (UniqueName: \"kubernetes.io/projected/1847b302-1b53-4f3d-a50e-521e22bdd784-kube-api-access-hkgg8\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.985054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984859    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd-console-oauth-config\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:51.985054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:51.984867    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2b588ec5-62c7-4ab2-9bdc-68f256f65fea-extensions-socket-volume\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:52.320819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.320774    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" path="/var/lib/kubelet/pods/1847b302-1b53-4f3d-a50e-521e22bdd784/volumes"
Jul 22 18:24:52.321151 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.321137    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" path="/var/lib/kubelet/pods/2b588ec5-62c7-4ab2-9bdc-68f256f65fea/volumes"
Jul 22 18:24:52.661267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.661226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m" event={"ID":"3f6a68df-04f0-4223-a91d-eca23e6ea3a0","Type":"ContainerStarted","Data":"2fa2a8070ddcda109d8fccedff55107b2248abc52d10e48c6ef29c790184e1d6"}
Jul 22 18:24:52.662559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.662535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f47d4cdf-9zbpb" event={"ID":"6cab0f3f-5465-4c16-9193-e0e5acc1b2dd","Type":"ContainerDied","Data":"722bc42d51436e892d31dd35b3c607880a726272e3ad60c7a0ed707fb9baa3a4"}
Jul 22 18:24:52.662559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.662545    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f47d4cdf-9zbpb"
Jul 22 18:24:52.662755 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.662623    2580 scope.go:117] "RemoveContainer" containerID="d5aaf69491833e9123ada0faa4724f10beccafc30e3224fc9424a62d915604b2"
Jul 22 18:24:52.663900 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.663886    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:52.665682 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.665660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc" event={"ID":"70e234f8-4190-4869-917a-8d185c61380a","Type":"ContainerStarted","Data":"e007ada11e19acd6ccdd5fee547da298872181d65a81463704419a6e2a63542d"}
Jul 22 18:24:52.665785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.665699    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc" event={"ID":"70e234f8-4190-4869-917a-8d185c61380a","Type":"ContainerStarted","Data":"9e5038a7c7ad1bf575a8941c7dd0c13e5029a638123b839a2b0f6ec0b2abf43f"}
Jul 22 18:24:52.665785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.665766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:24:52.667752 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.667730    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" event={"ID":"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b","Type":"ContainerStarted","Data":"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"}
Jul 22 18:24:52.667872 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.667774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" event={"ID":"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b","Type":"ContainerStarted","Data":"fa729286c5c769dca94e2990b49c3a3710bbb5127cb61f67bc6c703cc9a26f40"}
Jul 22 18:24:52.667872 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.667856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:52.669016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.669002    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:52.671704 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.671688    2580 scope.go:117] "RemoveContainer" containerID="ceed81b1d203a18bc1321f0597dc1109d4d3b38c0274a2dc7f280a9dd259405d"
Jul 22 18:24:52.677702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.677662    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-85h4m" podStartSLOduration=2.099856601 podStartE2EDuration="24.67765032s" totalImagesPullingTime="22.577793719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:52.676560181 +0000 UTC m=+646.892207602" watchObservedRunningTime="2026-07-22 18:24:52.67765032 +0000 UTC m=+646.893297723"
Jul 22 18:24:52.680184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.680169    2580 scope.go:117] "RemoveContainer" containerID="e6925e469cd66352c3d26fd6a1dec198439033a7994ce5355dcc7efaeb441909"
Jul 22 18:24:52.694370 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.694348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:24:52.700247 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.700221    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f47d4cdf-9zbpb"]
Jul 22 18:24:52.714078 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.714035    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" podStartSLOduration=14.714020475 podStartE2EDuration="14.714020475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:52.712904514 +0000 UTC m=+646.928551942" watchObservedRunningTime="2026-07-22 18:24:52.714020475 +0000 UTC m=+646.929667882"
Jul 22 18:24:52.727348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:52.727307    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc" podStartSLOduration=14.727293195 podStartE2EDuration="14.727293195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:24:52.726517062 +0000 UTC m=+646.942164456" watchObservedRunningTime="2026-07-22 18:24:52.727293195 +0000 UTC m=+646.942940598"
Jul 22 18:24:52.728570 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:52.728546    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-t7jfw\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="1847b302-1b53-4f3d-a50e-521e22bdd784" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-t7jfw"
Jul 22 18:24:52.730209 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:52.730183    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-gqprs\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="2b588ec5-62c7-4ab2-9bdc-68f256f65fea" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-gqprs"
Jul 22 18:24:54.321022 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:54.320991    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" path="/var/lib/kubelet/pods/6cab0f3f-5465-4c16-9193-e0e5acc1b2dd/volumes"
Jul 22 18:24:55.895766 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:55.895729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:55.896187 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:55.895953    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" podUID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" containerName="manager" containerID="cri-o://bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183" gracePeriod=10
Jul 22 18:24:56.143355 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.143327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:56.323275 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.323250    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume\") pod \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") "
Jul 22 18:24:56.323415 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.323287    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2\") pod \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\" (UID: \"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b\") "
Jul 22 18:24:56.323610 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.323586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" (UID: "c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:24:56.325329 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.325302    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2" (OuterVolumeSpecName: "kube-api-access-7pqk2") pod "c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" (UID: "c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b"). InnerVolumeSpecName "kube-api-access-7pqk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:24:56.424700 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.424663    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-extensions-socket-volume\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:56.424700 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.424696    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pqk2\" (UniqueName: \"kubernetes.io/projected/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b-kube-api-access-7pqk2\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:24:56.690130 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.690091    2580 generic.go:356] "Generic (PLEG): container finished" podID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" containerID="bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183" exitCode=0
Jul 22 18:24:56.690130 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.690131    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"
Jul 22 18:24:56.690449 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.690169    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" event={"ID":"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b","Type":"ContainerDied","Data":"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"}
Jul 22 18:24:56.690449 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.690229    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp" event={"ID":"c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b","Type":"ContainerDied","Data":"fa729286c5c769dca94e2990b49c3a3710bbb5127cb61f67bc6c703cc9a26f40"}
Jul 22 18:24:56.690449 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.690253    2580 scope.go:117] "RemoveContainer" containerID="bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"
Jul 22 18:24:56.698718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.698702    2580 scope.go:117] "RemoveContainer" containerID="bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"
Jul 22 18:24:56.698987 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:24:56.698967    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183\": container with ID starting with bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183 not found: ID does not exist" containerID="bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"
Jul 22 18:24:56.699046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.698996    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183"} err="failed to get container status \"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183\": rpc error: code = NotFound desc = could not find container \"bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183\": container with ID starting with bdd857cc52cba7ebc64adc8c8e3660a83da9f60bd7c27494396a8bcd72768183 not found: ID does not exist"
Jul 22 18:24:56.713685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.713657    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:56.718131 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:56.718105    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ssxlp"]
Jul 22 18:24:58.320712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:24:58.320676    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" path="/var/lib/kubelet/pods/c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b/volumes"
Jul 22 18:25:03.678667 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:03.678640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sxgdc"
Jul 22 18:25:14.172665 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.172626    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173102    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" containerName="manager"
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173122    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" containerName="manager"
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173148    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerName="console"
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173156    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerName="console"
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173262    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cab0f3f-5465-4c16-9193-e0e5acc1b2dd" containerName="console"
Jul 22 18:25:14.173293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.173276    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cb4c6e-f811-4aae-a78c-9f2b982a8e0b" containerName="manager"
Jul 22 18:25:14.328181 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.328151    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.330300 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.330265    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 22 18:25:14.333716 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.333695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:14.333845 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.333728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:14.369379 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.369356    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bpp\" (UniqueName: \"kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.369510 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.369455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.441601 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.441569    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:25:14.463406 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.463378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:25:14.463593 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.463463    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.467164 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.467145    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:25:14.470165 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.470144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.470285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.470181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f63327da-6ba2-426f-bb92-03bdba85bc14-config-file\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.470285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.470225    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8rl\" (UniqueName: \"kubernetes.io/projected/f63327da-6ba2-426f-bb92-03bdba85bc14-kube-api-access-cl8rl\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.470393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.470283    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bpp\" (UniqueName: \"kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.470899 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.470882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.482109 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.482089    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bpp\" (UniqueName: \"kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp\") pod \"limitador-limitador-64c94fd64f-z9skf\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.571275 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.571244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8rl\" (UniqueName: \"kubernetes.io/projected/f63327da-6ba2-426f-bb92-03bdba85bc14-kube-api-access-cl8rl\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.571427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.571319    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f63327da-6ba2-426f-bb92-03bdba85bc14-config-file\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.571891 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.571875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f63327da-6ba2-426f-bb92-03bdba85bc14-config-file\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.588480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.588452    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8rl\" (UniqueName: \"kubernetes.io/projected/f63327da-6ba2-426f-bb92-03bdba85bc14-kube-api-access-cl8rl\") pod \"limitador-limitador-54d6d988d5-wl4xl\" (UID: \"f63327da-6ba2-426f-bb92-03bdba85bc14\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.638505 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.638479    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:14.774415 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.774387    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:14.801896 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.801778    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:14.804395 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:25:14.804358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf984db_f749_4c4d_884d_6ead2d6b2298.slice/crio-3aee8809a48cf394f800d7841ce18dded8eaa6cba9cd2adbb67b7329f333a9c6 WatchSource:0}: Error finding container 3aee8809a48cf394f800d7841ce18dded8eaa6cba9cd2adbb67b7329f333a9c6: Status 404 returned error can't find the container with id 3aee8809a48cf394f800d7841ce18dded8eaa6cba9cd2adbb67b7329f333a9c6
Jul 22 18:25:14.909753 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.909727    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:25:14.911957 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:25:14.911924    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63327da_6ba2_426f_bb92_03bdba85bc14.slice/crio-f14b78351d98b0ccb3ce98328ca5242f750497bfac9494047a626327939bee1c WatchSource:0}: Error finding container f14b78351d98b0ccb3ce98328ca5242f750497bfac9494047a626327939bee1c: Status 404 returned error can't find the container with id f14b78351d98b0ccb3ce98328ca5242f750497bfac9494047a626327939bee1c
Jul 22 18:25:14.974519 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:14.974454    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:15.033488 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.033456    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:15.033641 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.033552    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:15.039228 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.039202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vls2r\""
Jul 22 18:25:15.074338 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.074308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkrkf\" (UniqueName: \"kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf\") pod \"authorino-58cd955675-rhkch\" (UID: \"732c0089-4d41-4679-8462-4d447634ac55\") " pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:15.174999 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.174967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkrkf\" (UniqueName: \"kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf\") pod \"authorino-58cd955675-rhkch\" (UID: \"732c0089-4d41-4679-8462-4d447634ac55\") " pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:15.183404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.183384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkrkf\" (UniqueName: \"kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf\") pod \"authorino-58cd955675-rhkch\" (UID: \"732c0089-4d41-4679-8462-4d447634ac55\") " pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:15.343209 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.343182    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:15.488534 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.488502    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:15.491588 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:25:15.491557    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod732c0089_4d41_4679_8462_4d447634ac55.slice/crio-a31129f594cc90d3a331f4754de0eed0a54a5491a6fd60348e3174a74df06e01 WatchSource:0}: Error finding container a31129f594cc90d3a331f4754de0eed0a54a5491a6fd60348e3174a74df06e01: Status 404 returned error can't find the container with id a31129f594cc90d3a331f4754de0eed0a54a5491a6fd60348e3174a74df06e01
Jul 22 18:25:15.774802 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.774755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl" event={"ID":"f63327da-6ba2-426f-bb92-03bdba85bc14","Type":"ContainerStarted","Data":"f14b78351d98b0ccb3ce98328ca5242f750497bfac9494047a626327939bee1c"}
Jul 22 18:25:15.775979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.775951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" event={"ID":"adf984db-f749-4c4d-884d-6ead2d6b2298","Type":"ContainerStarted","Data":"3aee8809a48cf394f800d7841ce18dded8eaa6cba9cd2adbb67b7329f333a9c6"}
Jul 22 18:25:15.776906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:15.776883    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhkch" event={"ID":"732c0089-4d41-4679-8462-4d447634ac55","Type":"ContainerStarted","Data":"a31129f594cc90d3a331f4754de0eed0a54a5491a6fd60348e3174a74df06e01"}
Jul 22 18:25:20.066302 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:20.066273    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:22.817100 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.817050    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhkch" event={"ID":"732c0089-4d41-4679-8462-4d447634ac55","Type":"ContainerStarted","Data":"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"}
Jul 22 18:25:22.817100 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.817071    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rhkch" podUID="732c0089-4d41-4679-8462-4d447634ac55" containerName="authorino" containerID="cri-o://37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52" gracePeriod=30
Jul 22 18:25:22.818712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.818682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl" event={"ID":"f63327da-6ba2-426f-bb92-03bdba85bc14","Type":"ContainerStarted","Data":"2fca722f21753429956e49dd2f308f0fd551f32346e57e6e11739fc656ea3850"}
Jul 22 18:25:22.819013 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.818767    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:22.820209 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.820181    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" event={"ID":"adf984db-f749-4c4d-884d-6ead2d6b2298","Type":"ContainerStarted","Data":"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"}
Jul 22 18:25:22.820352 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.820338    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:22.833528 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.833489    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rhkch" podStartSLOduration=2.1573844 podStartE2EDuration="8.833476302s" totalImagesPullingTime="6.676091902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:25:22.832884004 +0000 UTC m=+677.048531418" watchObservedRunningTime="2026-07-22 18:25:22.833476302 +0000 UTC m=+677.049123702"
Jul 22 18:25:22.848642 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.848602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl" podStartSLOduration=1.6074769789999999 podStartE2EDuration="8.848590002s" totalImagesPullingTime="7.241113023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:25:22.847120788 +0000 UTC m=+677.062768191" watchObservedRunningTime="2026-07-22 18:25:22.848590002 +0000 UTC m=+677.064237405"
Jul 22 18:25:22.862154 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:22.862110    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" podStartSLOduration=1.512357473 podStartE2EDuration="8.862096865s" totalImagesPullingTime="7.349739392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:25:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:25:22.860852207 +0000 UTC m=+677.076499614" watchObservedRunningTime="2026-07-22 18:25:22.862096865 +0000 UTC m=+677.077744293"
Jul 22 18:25:23.074082 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.074026    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:23.145796 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.145765    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkrkf\" (UniqueName: \"kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf\") pod \"732c0089-4d41-4679-8462-4d447634ac55\" (UID: \"732c0089-4d41-4679-8462-4d447634ac55\") "
Jul 22 18:25:23.147707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.147685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf" (OuterVolumeSpecName: "kube-api-access-qkrkf") pod "732c0089-4d41-4679-8462-4d447634ac55" (UID: "732c0089-4d41-4679-8462-4d447634ac55"). InnerVolumeSpecName "kube-api-access-qkrkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:25:23.246960 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.246918    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkrkf\" (UniqueName: \"kubernetes.io/projected/732c0089-4d41-4679-8462-4d447634ac55-kube-api-access-qkrkf\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:25:23.826103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.826048    2580 generic.go:356] "Generic (PLEG): container finished" podID="732c0089-4d41-4679-8462-4d447634ac55" containerID="37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52" exitCode=0
Jul 22 18:25:23.826103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.826090    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rhkch"
Jul 22 18:25:23.826647 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.826128    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhkch" event={"ID":"732c0089-4d41-4679-8462-4d447634ac55","Type":"ContainerDied","Data":"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"}
Jul 22 18:25:23.826647 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.826171    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rhkch" event={"ID":"732c0089-4d41-4679-8462-4d447634ac55","Type":"ContainerDied","Data":"a31129f594cc90d3a331f4754de0eed0a54a5491a6fd60348e3174a74df06e01"}
Jul 22 18:25:23.826647 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.826187    2580 scope.go:117] "RemoveContainer" containerID="37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"
Jul 22 18:25:23.835021 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.835005    2580 scope.go:117] "RemoveContainer" containerID="37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"
Jul 22 18:25:23.835253 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:25:23.835236    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52\": container with ID starting with 37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52 not found: ID does not exist" containerID="37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"
Jul 22 18:25:23.835313 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.835265    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52"} err="failed to get container status \"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52\": rpc error: code = NotFound desc = could not find container \"37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52\": container with ID starting with 37279b0798932c94529634878ed9ed7872a6e57b53eb2ee257609494daed9b52 not found: ID does not exist"
Jul 22 18:25:23.845516 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.845493    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:23.847983 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:23.847967    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rhkch"]
Jul 22 18:25:24.320721 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:24.320691    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="732c0089-4d41-4679-8462-4d447634ac55" path="/var/lib/kubelet/pods/732c0089-4d41-4679-8462-4d447634ac55/volumes"
Jul 22 18:25:33.828920 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:33.828888    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:33.829292 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:33.828958    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"
Jul 22 18:25:33.886779 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:33.886744    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:33.886942 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:33.886924    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" podUID="adf984db-f749-4c4d-884d-6ead2d6b2298" containerName="limitador" containerID="cri-o://ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758" gracePeriod=30
Jul 22 18:25:34.435581 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.435559    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:34.543085 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.542991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file\") pod \"adf984db-f749-4c4d-884d-6ead2d6b2298\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") "
Jul 22 18:25:34.543085 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.543038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7bpp\" (UniqueName: \"kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp\") pod \"adf984db-f749-4c4d-884d-6ead2d6b2298\" (UID: \"adf984db-f749-4c4d-884d-6ead2d6b2298\") "
Jul 22 18:25:34.543341 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.543320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file" (OuterVolumeSpecName: "config-file") pod "adf984db-f749-4c4d-884d-6ead2d6b2298" (UID: "adf984db-f749-4c4d-884d-6ead2d6b2298"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:25:34.545144 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.545123    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp" (OuterVolumeSpecName: "kube-api-access-t7bpp") pod "adf984db-f749-4c4d-884d-6ead2d6b2298" (UID: "adf984db-f749-4c4d-884d-6ead2d6b2298"). InnerVolumeSpecName "kube-api-access-t7bpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:25:34.644236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.644200    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/adf984db-f749-4c4d-884d-6ead2d6b2298-config-file\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:25:34.644236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.644232    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bpp\" (UniqueName: \"kubernetes.io/projected/adf984db-f749-4c4d-884d-6ead2d6b2298-kube-api-access-t7bpp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:25:34.876716 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.876632    2580 generic.go:356] "Generic (PLEG): container finished" podID="adf984db-f749-4c4d-884d-6ead2d6b2298" containerID="ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758" exitCode=0
Jul 22 18:25:34.876716 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.876672    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf"
Jul 22 18:25:34.876716 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.876710    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" event={"ID":"adf984db-f749-4c4d-884d-6ead2d6b2298","Type":"ContainerDied","Data":"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"}
Jul 22 18:25:34.877211 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.876759    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-z9skf" event={"ID":"adf984db-f749-4c4d-884d-6ead2d6b2298","Type":"ContainerDied","Data":"3aee8809a48cf394f800d7841ce18dded8eaa6cba9cd2adbb67b7329f333a9c6"}
Jul 22 18:25:34.877211 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.876783    2580 scope.go:117] "RemoveContainer" containerID="ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"
Jul 22 18:25:34.885371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.885353    2580 scope.go:117] "RemoveContainer" containerID="ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"
Jul 22 18:25:34.885614 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:25:34.885599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758\": container with ID starting with ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758 not found: ID does not exist" containerID="ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"
Jul 22 18:25:34.885662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.885622    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758"} err="failed to get container status \"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758\": rpc error: code = NotFound desc = could not find container \"ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758\": container with ID starting with ec6722e523273c65fe7a6fe00a770323c3a964514bc4ad2d6f421ba1d2c9b758 not found: ID does not exist"
Jul 22 18:25:34.896208 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.896186    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:34.898239 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:34.898219    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-z9skf"]
Jul 22 18:25:35.934153 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934118    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"]
Jul 22 18:25:35.934504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934449    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf984db-f749-4c4d-884d-6ead2d6b2298" containerName="limitador"
Jul 22 18:25:35.934504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934461    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf984db-f749-4c4d-884d-6ead2d6b2298" containerName="limitador"
Jul 22 18:25:35.934504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934488    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="732c0089-4d41-4679-8462-4d447634ac55" containerName="authorino"
Jul 22 18:25:35.934504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934495    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="732c0089-4d41-4679-8462-4d447634ac55" containerName="authorino"
Jul 22 18:25:35.934630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934571    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf984db-f749-4c4d-884d-6ead2d6b2298" containerName="limitador"
Jul 22 18:25:35.934630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.934581    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="732c0089-4d41-4679-8462-4d447634ac55" containerName="authorino"
Jul 22 18:25:35.937777 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.937756    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:35.944530 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.944499    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 22 18:25:35.945041 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.945018    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jf2fs\""
Jul 22 18:25:35.945156 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.945062    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 22 18:25:35.945309 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.945293    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 22 18:25:35.948735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:35.948713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"]
Jul 22 18:25:36.056189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.056157    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpr9\" (UniqueName: \"kubernetes.io/projected/f5560a20-0cc4-4e15-a1bb-7c2490209af0-kube-api-access-4bpr9\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.056356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.056207    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5560a20-0cc4-4e15-a1bb-7c2490209af0-data\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.156753 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.156724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5560a20-0cc4-4e15-a1bb-7c2490209af0-data\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.156960 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.156844    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpr9\" (UniqueName: \"kubernetes.io/projected/f5560a20-0cc4-4e15-a1bb-7c2490209af0-kube-api-access-4bpr9\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.157071 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.157051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f5560a20-0cc4-4e15-a1bb-7c2490209af0-data\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.165552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.165526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpr9\" (UniqueName: \"kubernetes.io/projected/f5560a20-0cc4-4e15-a1bb-7c2490209af0-kube-api-access-4bpr9\") pod \"postgres-69fb4594cf-6t9vv\" (UID: \"f5560a20-0cc4-4e15-a1bb-7c2490209af0\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.254123 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.254027    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:36.321032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.320997    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf984db-f749-4c4d-884d-6ead2d6b2298" path="/var/lib/kubelet/pods/adf984db-f749-4c4d-884d-6ead2d6b2298/volumes"
Jul 22 18:25:36.383519 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.383478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"]
Jul 22 18:25:36.386408 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:25:36.386383    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5560a20_0cc4_4e15_a1bb_7c2490209af0.slice/crio-4cf86997dee9f6f587041d68cead4fa537680e5c552915cbf301be3ced9a5382 WatchSource:0}: Error finding container 4cf86997dee9f6f587041d68cead4fa537680e5c552915cbf301be3ced9a5382: Status 404 returned error can't find the container with id 4cf86997dee9f6f587041d68cead4fa537680e5c552915cbf301be3ced9a5382
Jul 22 18:25:36.387731 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.387710    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:25:36.887390 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:36.887352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv" event={"ID":"f5560a20-0cc4-4e15-a1bb-7c2490209af0","Type":"ContainerStarted","Data":"4cf86997dee9f6f587041d68cead4fa537680e5c552915cbf301be3ced9a5382"}
Jul 22 18:25:42.920732 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:42.920633    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv" event={"ID":"f5560a20-0cc4-4e15-a1bb-7c2490209af0","Type":"ContainerStarted","Data":"945db39af8bc06ee4ef9a0b371e7c70f1858dfafa46bfc6b25d7ff072f476ef3"}
Jul 22 18:25:42.920732 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:42.920705    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:42.939347 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:42.939250    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv" podStartSLOduration=1.758225127 podStartE2EDuration="7.939233277s" totalImagesPullingTime="6.18100815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:25:42.938195767 +0000 UTC m=+697.153843172" watchObservedRunningTime="2026-07-22 18:25:42.939233277 +0000 UTC m=+697.154880685"
Jul 22 18:25:48.954891 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:48.954862    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-6t9vv"
Jul 22 18:25:49.512969 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.512938    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2wvgz"]
Jul 22 18:25:49.520477 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.520452    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:49.523042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.523021    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vls2r\""
Jul 22 18:25:49.524402 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.524365    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2wvgz"]
Jul 22 18:25:49.673976 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.673940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkj6\" (UniqueName: \"kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6\") pod \"authorino-5d4b67c446-2wvgz\" (UID: \"a81c4079-9ff3-4089-adf9-efd1ee35b096\") " pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:49.749642 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.749609    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2wvgz"]
Jul 22 18:25:49.749888 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:25:49.749868    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rxkj6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-2wvgz" podUID="a81c4079-9ff3-4089-adf9-efd1ee35b096"
Jul 22 18:25:49.775209 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.775135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkj6\" (UniqueName: \"kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6\") pod \"authorino-5d4b67c446-2wvgz\" (UID: \"a81c4079-9ff3-4089-adf9-efd1ee35b096\") " pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:49.778711 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.778690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-776c5b577-tmdj5"]
Jul 22 18:25:49.785017 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.784990    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkj6\" (UniqueName: \"kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6\") pod \"authorino-5d4b67c446-2wvgz\" (UID: \"a81c4079-9ff3-4089-adf9-efd1ee35b096\") " pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:49.831915 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.831882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-776c5b577-tmdj5"]
Jul 22 18:25:49.832048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.832028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:49.951384 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.951353    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:49.976647 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.976613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrkr\" (UniqueName: \"kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr\") pod \"authorino-776c5b577-tmdj5\" (UID: \"2980b7c5-d146-42bb-8c46-d293620565c5\") " pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:49.980585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.980553    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-776c5b577-tmdj5"]
Jul 22 18:25:49.980783 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:25:49.980766    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ctrkr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-776c5b577-tmdj5" podUID="2980b7c5-d146-42bb-8c46-d293620565c5"
Jul 22 18:25:49.981115 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:49.981098    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:50.012276 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.012248    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:25:50.022557 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.022528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.023334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.023308    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:25:50.027733 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.027673    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 22 18:25:50.077937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.077907    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxkj6\" (UniqueName: \"kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6\") pod \"a81c4079-9ff3-4089-adf9-efd1ee35b096\" (UID: \"a81c4079-9ff3-4089-adf9-efd1ee35b096\") "
Jul 22 18:25:50.078087 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.078061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrkr\" (UniqueName: \"kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr\") pod \"authorino-776c5b577-tmdj5\" (UID: \"2980b7c5-d146-42bb-8c46-d293620565c5\") " pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:50.080008 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.079979    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6" (OuterVolumeSpecName: "kube-api-access-rxkj6") pod "a81c4079-9ff3-4089-adf9-efd1ee35b096" (UID: "a81c4079-9ff3-4089-adf9-efd1ee35b096"). InnerVolumeSpecName "kube-api-access-rxkj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:25:50.091077 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.091053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrkr\" (UniqueName: \"kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr\") pod \"authorino-776c5b577-tmdj5\" (UID: \"2980b7c5-d146-42bb-8c46-d293620565c5\") " pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:50.178630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.178590    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5rv\" (UniqueName: \"kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.178794 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.178642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.178794 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.178764    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkj6\" (UniqueName: \"kubernetes.io/projected/a81c4079-9ff3-4089-adf9-efd1ee35b096-kube-api-access-rxkj6\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:25:50.280179 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.280090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5rv\" (UniqueName: \"kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.280179 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.280129    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.282516 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.282487    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.288403 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.288379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5rv\" (UniqueName: \"kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv\") pod \"authorino-5dc97789cc-bn7kd\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") " pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.331918 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.331894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:25:50.462189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.462163    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:25:50.464432 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:25:50.464407    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5c00c2e_4737_4c6f_8c03_7f53073374d0.slice/crio-a45ae9a9a02132f66097ea3d44fe443a0dd1930b1ed1ca26aea32c683b0e183f WatchSource:0}: Error finding container a45ae9a9a02132f66097ea3d44fe443a0dd1930b1ed1ca26aea32c683b0e183f: Status 404 returned error can't find the container with id a45ae9a9a02132f66097ea3d44fe443a0dd1930b1ed1ca26aea32c683b0e183f
Jul 22 18:25:50.958623 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.958582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" event={"ID":"c5c00c2e-4737-4c6f-8c03-7f53073374d0","Type":"ContainerStarted","Data":"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"}
Jul 22 18:25:50.958901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.958633    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:50.958901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.958599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2wvgz"
Jul 22 18:25:50.958901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.958641    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" event={"ID":"c5c00c2e-4737-4c6f-8c03-7f53073374d0","Type":"ContainerStarted","Data":"a45ae9a9a02132f66097ea3d44fe443a0dd1930b1ed1ca26aea32c683b0e183f"}
Jul 22 18:25:50.967105 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.967078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:50.974328 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:50.974289    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" podStartSLOduration=1.628141573 podStartE2EDuration="1.974274568s" totalImagesPullingTime="346.132995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:25:50.97412844 +0000 UTC m=+705.189775843" watchObservedRunningTime="2026-07-22 18:25:50.974274568 +0000 UTC m=+705.189921970"
Jul 22 18:25:51.000519 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.000485    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2wvgz"]
Jul 22 18:25:51.005164 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.005139    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2wvgz"]
Jul 22 18:25:51.086446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.086362    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctrkr\" (UniqueName: \"kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr\") pod \"2980b7c5-d146-42bb-8c46-d293620565c5\" (UID: \"2980b7c5-d146-42bb-8c46-d293620565c5\") "
Jul 22 18:25:51.088456 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.088422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr" (OuterVolumeSpecName: "kube-api-access-ctrkr") pod "2980b7c5-d146-42bb-8c46-d293620565c5" (UID: "2980b7c5-d146-42bb-8c46-d293620565c5"). InnerVolumeSpecName "kube-api-access-ctrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:25:51.187792 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.187758    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctrkr\" (UniqueName: \"kubernetes.io/projected/2980b7c5-d146-42bb-8c46-d293620565c5-kube-api-access-ctrkr\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:25:51.963312 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.963277    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-776c5b577-tmdj5"
Jul 22 18:25:51.995656 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:51.995629    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-776c5b577-tmdj5"]
Jul 22 18:25:52.001388 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:52.001362    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-776c5b577-tmdj5"]
Jul 22 18:25:52.319900 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:52.319824    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2980b7c5-d146-42bb-8c46-d293620565c5" path="/var/lib/kubelet/pods/2980b7c5-d146-42bb-8c46-d293620565c5/volumes"
Jul 22 18:25:52.320056 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:25:52.320044    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a81c4079-9ff3-4089-adf9-efd1ee35b096" path="/var/lib/kubelet/pods/a81c4079-9ff3-4089-adf9-efd1ee35b096/volumes"
Jul 22 18:26:32.099948 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:32.099911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:26:47.592935 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.592900    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"]
Jul 22 18:26:47.595563 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.595540    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.598410 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.598384    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 22 18:26:47.599559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.599530    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 22 18:26:47.599680 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.599533    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 22 18:26:47.599680 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.599567    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vrkz\""
Jul 22 18:26:47.608728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.608703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"]
Jul 22 18:26:47.753685 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.753911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753727    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f988ac90-ad1f-47fb-845e-9f79f68d937f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.753911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.753911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753781    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.753911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrnl\" (UniqueName: \"kubernetes.io/projected/f988ac90-ad1f-47fb-845e-9f79f68d937f-kube-api-access-xdrnl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.754062 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.753910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.854910 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.854833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.854910 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.854906    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.854985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f988ac90-ad1f-47fb-845e-9f79f68d937f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855024    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrnl\" (UniqueName: \"kubernetes.io/projected/f988ac90-ad1f-47fb-845e-9f79f68d937f-kube-api-access-xdrnl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855301 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855267    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855352 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.855405 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.855383    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.857063 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.857031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f988ac90-ad1f-47fb-845e-9f79f68d937f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.857348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.857332    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f988ac90-ad1f-47fb-845e-9f79f68d937f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.862558 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.862535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrnl\" (UniqueName: \"kubernetes.io/projected/f988ac90-ad1f-47fb-845e-9f79f68d937f-kube-api-access-xdrnl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv\" (UID: \"f988ac90-ad1f-47fb-845e-9f79f68d937f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:47.907282 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:47.907241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:48.039972 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:48.039703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"]
Jul 22 18:26:48.042769 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:26:48.042739    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf988ac90_ad1f_47fb_845e_9f79f68d937f.slice/crio-93d33e978b22be0badd7b22cf89edd35731895344120c7bfa574fe6d31b62c53 WatchSource:0}: Error finding container 93d33e978b22be0badd7b22cf89edd35731895344120c7bfa574fe6d31b62c53: Status 404 returned error can't find the container with id 93d33e978b22be0badd7b22cf89edd35731895344120c7bfa574fe6d31b62c53
Jul 22 18:26:48.073765 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:48.073717    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:26:48.215901 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:48.215867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv" event={"ID":"f988ac90-ad1f-47fb-845e-9f79f68d937f","Type":"ContainerStarted","Data":"93d33e978b22be0badd7b22cf89edd35731895344120c7bfa574fe6d31b62c53"}
Jul 22 18:26:50.227396 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:50.227356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv" event={"ID":"f988ac90-ad1f-47fb-845e-9f79f68d937f","Type":"ContainerStarted","Data":"0e4a11f99de3df7d5e9099c90626112a285fc0798f99745d375bada4c24ff18e"}
Jul 22 18:26:50.243978 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:50.243931    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv" podStartSLOduration=2.001900669 podStartE2EDuration="3.243913258s" totalImagesPullingTime="1.242012589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:26:50.242999097 +0000 UTC m=+764.458646501" watchObservedRunningTime="2026-07-22 18:26:50.243913258 +0000 UTC m=+764.459560661"
Jul 22 18:26:57.275415 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.275227    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"]
Jul 22 18:26:57.279325 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.279296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.281588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.281563    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 22 18:26:57.288189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.288165    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"]
Jul 22 18:26:57.442381 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2tcl\" (UniqueName: \"kubernetes.io/projected/482f9edf-b110-4b38-b93e-152f865f0f92-kube-api-access-b2tcl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.442381 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.442588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.442588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442510    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.442588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442541    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.442588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.442572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482f9edf-b110-4b38-b93e-152f865f0f92-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543175    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2tcl\" (UniqueName: \"kubernetes.io/projected/482f9edf-b110-4b38-b93e-152f865f0f92-kube-api-access-b2tcl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543279    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543378    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482f9edf-b110-4b38-b93e-152f865f0f92-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543597 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543581    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543725    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.543837 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.543735    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.545489 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.545467    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/482f9edf-b110-4b38-b93e-152f865f0f92-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.545827 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.545790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/482f9edf-b110-4b38-b93e-152f865f0f92-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.551206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.551178    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2tcl\" (UniqueName: \"kubernetes.io/projected/482f9edf-b110-4b38-b93e-152f865f0f92-kube-api-access-b2tcl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-ckvps\" (UID: \"482f9edf-b110-4b38-b93e-152f865f0f92\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.591531 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.591495    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:26:57.716515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.716480    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"]
Jul 22 18:26:57.719085 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:26:57.719057    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482f9edf_b110_4b38_b93e_152f865f0f92.slice/crio-d3a97d1457e51311ba96e75a26eb16d9abc0b3b34c2b183c5d0c64d98002b319 WatchSource:0}: Error finding container d3a97d1457e51311ba96e75a26eb16d9abc0b3b34c2b183c5d0c64d98002b319: Status 404 returned error can't find the container with id d3a97d1457e51311ba96e75a26eb16d9abc0b3b34c2b183c5d0c64d98002b319
Jul 22 18:26:57.907779 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.907695    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:57.907941 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.907791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:57.920258 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:57.920220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:58.077431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:58.077387    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:26:58.266170 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:58.266133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps" event={"ID":"482f9edf-b110-4b38-b93e-152f865f0f92","Type":"ContainerStarted","Data":"7a76a71db2fd09797b47dc79c606f578c06306007b995ddbfb15444c816a6ee9"}
Jul 22 18:26:58.266336 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:58.266190    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps" event={"ID":"482f9edf-b110-4b38-b93e-152f865f0f92","Type":"ContainerStarted","Data":"d3a97d1457e51311ba96e75a26eb16d9abc0b3b34c2b183c5d0c64d98002b319"}
Jul 22 18:26:58.277565 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:58.277537    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv"
Jul 22 18:26:58.283981 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:26:58.283935    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps" podStartSLOduration=1.030946192 podStartE2EDuration="1.283920807s" totalImagesPullingTime="252.974615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:26:58.282028463 +0000 UTC m=+772.497675866" watchObservedRunningTime="2026-07-22 18:26:58.283920807 +0000 UTC m=+772.499568210"
Jul 22 18:27:05.574977 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:05.574936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:27:07.592502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:07.592467    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:27:07.592944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:07.592520    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:27:07.605321 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:07.605294    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:27:08.327339 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:08.327309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-ckvps"
Jul 22 18:27:11.178775 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:11.178734    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:27:20.363941 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:20.363908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:27:31.786389 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:27:31.786309    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:28:22.021248 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.021213    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:28:22.021737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.021425    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" podUID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" containerName="authorino" containerID="cri-o://6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6" gracePeriod=30
Jul 22 18:28:22.270904 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.270883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:28:22.324178 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.324105    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert\") pod \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") "
Jul 22 18:28:22.324178 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.324163    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr5rv\" (UniqueName: \"kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv\") pod \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\" (UID: \"c5c00c2e-4737-4c6f-8c03-7f53073374d0\") "
Jul 22 18:28:22.326104 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.326075    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv" (OuterVolumeSpecName: "kube-api-access-vr5rv") pod "c5c00c2e-4737-4c6f-8c03-7f53073374d0" (UID: "c5c00c2e-4737-4c6f-8c03-7f53073374d0"). InnerVolumeSpecName "kube-api-access-vr5rv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:28:22.333970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.333949    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c5c00c2e-4737-4c6f-8c03-7f53073374d0" (UID: "c5c00c2e-4737-4c6f-8c03-7f53073374d0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:28:22.425206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.425173    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c5c00c2e-4737-4c6f-8c03-7f53073374d0-tls-cert\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:28:22.425206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.425200    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5rv\" (UniqueName: \"kubernetes.io/projected/c5c00c2e-4737-4c6f-8c03-7f53073374d0-kube-api-access-vr5rv\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:28:22.657830 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.657737    2580 generic.go:356] "Generic (PLEG): container finished" podID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" containerID="6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6" exitCode=0
Jul 22 18:28:22.657830 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.657781    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc97789cc-bn7kd"
Jul 22 18:28:22.658003 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.657843    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" event={"ID":"c5c00c2e-4737-4c6f-8c03-7f53073374d0","Type":"ContainerDied","Data":"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"}
Jul 22 18:28:22.658003 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.657887    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc97789cc-bn7kd" event={"ID":"c5c00c2e-4737-4c6f-8c03-7f53073374d0","Type":"ContainerDied","Data":"a45ae9a9a02132f66097ea3d44fe443a0dd1930b1ed1ca26aea32c683b0e183f"}
Jul 22 18:28:22.658003 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.657905    2580 scope.go:117] "RemoveContainer" containerID="6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"
Jul 22 18:28:22.668948 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.668928    2580 scope.go:117] "RemoveContainer" containerID="6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"
Jul 22 18:28:22.669195 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:28:22.669178    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6\": container with ID starting with 6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6 not found: ID does not exist" containerID="6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"
Jul 22 18:28:22.669242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.669206    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6"} err="failed to get container status \"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6\": rpc error: code = NotFound desc = could not find container \"6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6\": container with ID starting with 6969fd0421dfa5c118363a9a896da8efd662ce7d2abab76f92d2623e46af3fc6 not found: ID does not exist"
Jul 22 18:28:22.684929 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.684907    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:28:22.686502 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:22.686483    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc97789cc-bn7kd"]
Jul 22 18:28:24.320274 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:24.320244    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" path="/var/lib/kubelet/pods/c5c00c2e-4737-4c6f-8c03-7f53073374d0/volumes"
Jul 22 18:28:51.476929 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:28:51.476885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:29:00.776426 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:29:00.776344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:29:12.081725 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:29:12.081689    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:30:00.146561 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.146529    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"]
Jul 22 18:30:00.147042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.146921    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" containerName="authorino"
Jul 22 18:30:00.147042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.146933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" containerName="authorino"
Jul 22 18:30:00.147042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.146984    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5c00c2e-4737-4c6f-8c03-7f53073374d0" containerName="authorino"
Jul 22 18:30:00.150036 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.150020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"
Jul 22 18:30:00.152670 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.152645    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:30:00.157859 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.157838    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"]
Jul 22 18:30:00.160862 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.160844    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"
Jul 22 18:30:00.294075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:00.294049    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"]
Jul 22 18:30:00.295877 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:30:00.295845    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10cd516_f596_436d_8f3a_dc8e98b0cade.slice/crio-0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5 WatchSource:0}: Error finding container 0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5: Status 404 returned error can't find the container with id 0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5
Jul 22 18:30:01.108315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:01.108278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6" event={"ID":"d10cd516-f596-436d-8f3a-dc8e98b0cade","Type":"ContainerStarted","Data":"0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5"}
Jul 22 18:30:05.130746 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:05.130715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6" event={"ID":"d10cd516-f596-436d-8f3a-dc8e98b0cade","Type":"ContainerStarted","Data":"cb2a3110ee1874da776b83d64f16c5ed21a1185954afeb60dbfe8139e3c11d4d"}
Jul 22 18:30:05.144700 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:05.144643    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6" podStartSLOduration=1.201624506 podStartE2EDuration="5.144626228s" totalImagesPullingTime="3.943001722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:30:05.144155028 +0000 UTC m=+959.359802432" watchObservedRunningTime="2026-07-22 18:30:05.144626228 +0000 UTC m=+959.360273628"
Jul 22 18:30:06.137140 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:06.137103    2580 generic.go:356] "Generic (PLEG): container finished" podID="d10cd516-f596-436d-8f3a-dc8e98b0cade" containerID="cb2a3110ee1874da776b83d64f16c5ed21a1185954afeb60dbfe8139e3c11d4d" exitCode=0
Jul 22 18:30:06.137522 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:06.137192    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6" event={"ID":"d10cd516-f596-436d-8f3a-dc8e98b0cade","Type":"ContainerDied","Data":"cb2a3110ee1874da776b83d64f16c5ed21a1185954afeb60dbfe8139e3c11d4d"}
Jul 22 18:30:07.271207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:07.271182    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"
Jul 22 18:30:08.147170 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:08.147144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6"
Jul 22 18:30:08.147339 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:08.147182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745750-hznl6" event={"ID":"d10cd516-f596-436d-8f3a-dc8e98b0cade","Type":"ContainerDied","Data":"0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5"}
Jul 22 18:30:08.147339 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:08.147229    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0be2826e56dcd02a6a32c4e452f0cc78d54bf090330e0bf8549d7c2126814ac5"
Jul 22 18:30:54.218443 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:30:54.218404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:31:06.167092 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:31:06.167057    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:31:46.171434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:31:46.171395    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:31:55.269435 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:31:55.269394    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:05.669195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:05.669115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:13.482991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:13.482961    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:24.476931 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:24.476895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:34.882191 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:34.882159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:45.871370 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:45.871330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:32:55.614152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:32:55.614114    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:33:18.103336 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:33:18.103296    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:33:30.089042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:33:30.088969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:33:53.881089 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:33:53.881046    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:34:08.067231 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:34:08.067197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:34:19.577196 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:34:19.577161    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:34:31.171205 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:34:31.171164    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:34:41.375818 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:34:41.375768    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:34:52.982662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:34:52.982623    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:36:46.480153 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:36:46.480074    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:36:56.067947 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:36:56.067913    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:37:09.578880 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:37:09.578842    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:37:20.174936 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:37:20.174899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:37:31.581169 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:37:31.581132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:37:41.071482 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:37:41.071435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:37:51.476178 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:37:51.476136    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:38:01.269770 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:38:01.269679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:38:11.890881 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:38:11.890846    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:38:21.477754 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:38:21.477718    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:38:33.479541 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:38:33.479510    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:38:58.681577 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:38:58.681541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:39:05.880128 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:05.880085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:39:09.599058 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.599024    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:39:09.599420 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.599395    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d10cd516-f596-436d-8f3a-dc8e98b0cade" containerName="cleanup"
Jul 22 18:39:09.599420 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.599406    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d10cd516-f596-436d-8f3a-dc8e98b0cade" containerName="cleanup"
Jul 22 18:39:09.599499 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.599494    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d10cd516-f596-436d-8f3a-dc8e98b0cade" containerName="cleanup"
Jul 22 18:39:09.602378 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.602361    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.604743 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.604722    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0ed5c03b-kserve-self-signed-certs\""
Jul 22 18:39:09.618023 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.617997    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:39:09.685982 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.685941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.686195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.686011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.686195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.686046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.686195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.686077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.686195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.686100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.686195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.686165    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787245 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787418    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787786 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787751    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787952 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787767    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.787952 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.787863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.789540 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.789522    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.789782 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.789766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.796280 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.796257    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw\") pod \"e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") " pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:09.914886 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:09.914768    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:10.047696 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:10.047657    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:39:10.050487 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:39:10.050459    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6c5e66d_7183_4502_9c13_b11f7c07cb8a.slice/crio-78770c26923da3709715fc661e598080e736780c8a0c8d6e4a16cba0e47c9388 WatchSource:0}: Error finding container 78770c26923da3709715fc661e598080e736780c8a0c8d6e4a16cba0e47c9388: Status 404 returned error can't find the container with id 78770c26923da3709715fc661e598080e736780c8a0c8d6e4a16cba0e47c9388
Jul 22 18:39:10.626967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:10.626930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" event={"ID":"a6c5e66d-7183-4502-9c13-b11f7c07cb8a","Type":"ContainerStarted","Data":"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"}
Jul 22 18:39:10.626967 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:10.626972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" event={"ID":"a6c5e66d-7183-4502-9c13-b11f7c07cb8a","Type":"ContainerStarted","Data":"78770c26923da3709715fc661e598080e736780c8a0c8d6e4a16cba0e47c9388"}
Jul 22 18:39:10.642906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:10.642854    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" podStartSLOduration=1.642837971 podStartE2EDuration="1.642837971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:39:10.642061901 +0000 UTC m=+1504.857709316" watchObservedRunningTime="2026-07-22 18:39:10.642837971 +0000 UTC m=+1504.858485373"
Jul 22 18:39:19.915798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:19.915716    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:19.915798 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:19.915765    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:19.928314 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:19.928291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:20.684919 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:20.684890    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:39:21.776380 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:21.776341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:39:24.561936 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.561906    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:39:24.565263 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.565246    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.567934 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.567913    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0ed5c03b-kserve-self-signed-certs\""
Jul 22 18:39:24.583897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.583872    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:39:24.620816 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.620961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.620961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.620961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7s5\" (UniqueName: \"kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.620961 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.621100 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.620963    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721479 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721479 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721489    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7s5\" (UniqueName: \"kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.721707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.721625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.722110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.722085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.722110 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.722100    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.722244 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.722152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.723886 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.723863    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.724117 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.724103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.730523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.730497    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7s5\" (UniqueName: \"kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5\") pod \"e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") " pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:24.874596 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:24.874514    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:25.013217 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:25.013186    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:39:25.019056 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:39:25.018963    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod086dd144_eae6_424b_bb5c_cee51018b8e2.slice/crio-8027aec552f288f9ebe4df64a6ace05a32ff5bf84bc062d22522253b5ec881a5 WatchSource:0}: Error finding container 8027aec552f288f9ebe4df64a6ace05a32ff5bf84bc062d22522253b5ec881a5: Status 404 returned error can't find the container with id 8027aec552f288f9ebe4df64a6ace05a32ff5bf84bc062d22522253b5ec881a5
Jul 22 18:39:25.698351 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:25.698307    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" event={"ID":"086dd144-eae6-424b-bb5c-cee51018b8e2","Type":"ContainerStarted","Data":"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"}
Jul 22 18:39:25.698735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:25.698362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" event={"ID":"086dd144-eae6-424b-bb5c-cee51018b8e2","Type":"ContainerStarted","Data":"8027aec552f288f9ebe4df64a6ace05a32ff5bf84bc062d22522253b5ec881a5"}
Jul 22 18:39:25.716633 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:25.716584    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" podStartSLOduration=1.716569488 podStartE2EDuration="1.716569488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:39:25.715676246 +0000 UTC m=+1519.931323649" watchObservedRunningTime="2026-07-22 18:39:25.716569488 +0000 UTC m=+1519.932216891"
Jul 22 18:39:34.875692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:34.875611    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:34.875692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:34.875665    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:34.888383 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:34.888358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:35.755922 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:35.755891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:39:36.904284 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:36.904245    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:39:46.860056 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:46.860015    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:39:48.263460 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:48.263422    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:39:51.773993 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:39:51.773949    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:01.880466 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:01.880407    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" podUID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" containerName="main" containerID="cri-o://324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12" gracePeriod=45
Jul 22 18:40:02.130173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.130149    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:40:02.264275 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264238    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264430    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264461    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264600 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264513    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264600 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264600 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264578    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm\") pod \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\" (UID: \"a6c5e66d-7183-4502-9c13-b11f7c07cb8a\") "
Jul 22 18:40:02.264788 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:02.264788 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264779    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home" (OuterVolumeSpecName: "home") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:02.264903 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264792    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:02.264940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264930    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.264986 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264950    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.264986 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.264964    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.266584 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.266555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm" (OuterVolumeSpecName: "dshm") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:02.266666 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.266590    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:40:02.266666 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.266651    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw" (OuterVolumeSpecName: "kube-api-access-bk2lw") pod "a6c5e66d-7183-4502-9c13-b11f7c07cb8a" (UID: "a6c5e66d-7183-4502-9c13-b11f7c07cb8a"). InnerVolumeSpecName "kube-api-access-bk2lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:40:02.366249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.366215    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.366249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.366248    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.366460 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.366260    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk2lw\" (UniqueName: \"kubernetes.io/projected/a6c5e66d-7183-4502-9c13-b11f7c07cb8a-kube-api-access-bk2lw\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:02.876987 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.876950    2580 generic.go:356] "Generic (PLEG): container finished" podID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" containerID="324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12" exitCode=0
Jul 22 18:40:02.877189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.876998    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"
Jul 22 18:40:02.877189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.877040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" event={"ID":"a6c5e66d-7183-4502-9c13-b11f7c07cb8a","Type":"ContainerDied","Data":"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"}
Jul 22 18:40:02.877189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.877097    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7" event={"ID":"a6c5e66d-7183-4502-9c13-b11f7c07cb8a","Type":"ContainerDied","Data":"78770c26923da3709715fc661e598080e736780c8a0c8d6e4a16cba0e47c9388"}
Jul 22 18:40:02.877189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.877121    2580 scope.go:117] "RemoveContainer" containerID="324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"
Jul 22 18:40:02.885696 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.885552    2580 scope.go:117] "RemoveContainer" containerID="324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"
Jul 22 18:40:02.885918 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:40:02.885844    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12\": container with ID starting with 324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12 not found: ID does not exist" containerID="324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"
Jul 22 18:40:02.885918 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.885871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12"} err="failed to get container status \"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12\": rpc error: code = NotFound desc = could not find container \"324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12\": container with ID starting with 324aa1abbbf59c9af8498fe4767f55568051a01882f47a3d9762449029d6bf12 not found: ID does not exist"
Jul 22 18:40:02.893023 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.892999    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:40:02.896801 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:02.896778    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0ed5c03b-kserve-6f6d87978c-xrqt7"]
Jul 22 18:40:03.283605 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.283562    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" podUID="086dd144-eae6-424b-bb5c-cee51018b8e2" containerName="main" containerID="cri-o://b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9" gracePeriod=45
Jul 22 18:40:03.540673 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.540615    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:40:03.676897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.676856    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.676924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.676960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.676999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr7s5\" (UniqueName: \"kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677031    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache\") pod \"086dd144-eae6-424b-bb5c-cee51018b8e2\" (UID: \"086dd144-eae6-424b-bb5c-cee51018b8e2\") "
Jul 22 18:40:03.677431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677201    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home" (OuterVolumeSpecName: "home") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:03.677431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677341    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.677431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677386    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:03.677431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.677410    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:03.679080 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.679057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm" (OuterVolumeSpecName: "dshm") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:40:03.679080 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.679072    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:40:03.679209 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.679185    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5" (OuterVolumeSpecName: "kube-api-access-rr7s5") pod "086dd144-eae6-424b-bb5c-cee51018b8e2" (UID: "086dd144-eae6-424b-bb5c-cee51018b8e2"). InnerVolumeSpecName "kube-api-access-rr7s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:40:03.778493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.778450    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/086dd144-eae6-424b-bb5c-cee51018b8e2-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.778493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.778483    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.778493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.778493    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7s5\" (UniqueName: \"kubernetes.io/projected/086dd144-eae6-424b-bb5c-cee51018b8e2-kube-api-access-rr7s5\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.778493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.778502    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.778742 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.778512    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/086dd144-eae6-424b-bb5c-cee51018b8e2-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:40:03.885046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.884967    2580 generic.go:356] "Generic (PLEG): container finished" podID="086dd144-eae6-424b-bb5c-cee51018b8e2" containerID="b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9" exitCode=0
Jul 22 18:40:03.885046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.885015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"
Jul 22 18:40:03.885247 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.885049    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" event={"ID":"086dd144-eae6-424b-bb5c-cee51018b8e2","Type":"ContainerDied","Data":"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"}
Jul 22 18:40:03.885247 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.885088    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s" event={"ID":"086dd144-eae6-424b-bb5c-cee51018b8e2","Type":"ContainerDied","Data":"8027aec552f288f9ebe4df64a6ace05a32ff5bf84bc062d22522253b5ec881a5"}
Jul 22 18:40:03.885247 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.885105    2580 scope.go:117] "RemoveContainer" containerID="b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"
Jul 22 18:40:03.895678 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.895642    2580 scope.go:117] "RemoveContainer" containerID="b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"
Jul 22 18:40:03.896173 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:40:03.896148    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9\": container with ID starting with b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9 not found: ID does not exist" containerID="b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"
Jul 22 18:40:03.896228 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.896186    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9"} err="failed to get container status \"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9\": rpc error: code = NotFound desc = could not find container \"b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9\": container with ID starting with b5d1f7580af823d9bcffdc5b5d1ce825396a073162e0b531c4e7b35cc956b8d9 not found: ID does not exist"
Jul 22 18:40:03.904902 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.904876    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:40:03.908776 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:03.908754    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0ed5c03b-kserve-59c8b6d79f-5cj2s"]
Jul 22 18:40:04.322085 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:04.322051    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="086dd144-eae6-424b-bb5c-cee51018b8e2" path="/var/lib/kubelet/pods/086dd144-eae6-424b-bb5c-cee51018b8e2/volumes"
Jul 22 18:40:04.322439 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:04.322427    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" path="/var/lib/kubelet/pods/a6c5e66d-7183-4502-9c13-b11f7c07cb8a/volumes"
Jul 22 18:40:06.064375 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:06.064337    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:15.882052 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:15.882016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:28.364129 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:28.364098    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:39.270486 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:39.270442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:48.574573 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:48.574539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:40:59.070720 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:40:59.070640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:07.274182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:07.274146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:17.083629 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:17.083593    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:30.183606 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:30.183569    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:38.983545 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:38.983500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:49.269124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:49.269086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:41:59.467507 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:41:59.467470    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:42:12.174560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:42:12.174521    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:42:20.874655 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:42:20.874622    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:42:33.688642 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:42:33.688565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:42:45.979113 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:42:45.979082    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:42:58.972261 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:42:58.972225    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:43:07.973271 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:07.973233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:43:29.281700 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:29.281661    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:43:37.933735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.933701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:43:37.934132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934118    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" containerName="main"
Jul 22 18:43:37.934132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934133    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" containerName="main"
Jul 22 18:43:37.934204 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934145    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086dd144-eae6-424b-bb5c-cee51018b8e2" containerName="main"
Jul 22 18:43:37.934204 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="086dd144-eae6-424b-bb5c-cee51018b8e2" containerName="main"
Jul 22 18:43:37.934263 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934209    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="086dd144-eae6-424b-bb5c-cee51018b8e2" containerName="main"
Jul 22 18:43:37.934263 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.934220    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c5e66d-7183-4502-9c13-b11f7c07cb8a" containerName="main"
Jul 22 18:43:37.937155 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.937138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:37.939834 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.939791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:43:37.939945 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.939890    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-357ba574-serving-cert\""
Jul 22 18:43:37.950026 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:37.950008    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:43:38.044897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.044869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.045016 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.044906    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgc4g\" (UniqueName: \"kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.045057 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.045031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.145370 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.145345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.145481 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.145381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.145481 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.145401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgc4g\" (UniqueName: \"kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.145549 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:43:38.145489    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-serving-cert: secret "maas-api-e2e-ait-357ba574-serving-cert" not found
Jul 22 18:43:38.145583 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:43:38.145553    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls podName:4a765418-feb5-41d7-a182-2c22263b21bc nodeName:}" failed. No retries permitted until 2026-07-22 18:43:38.645535169 +0000 UTC m=+1772.861182553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls") pod "maas-api-e2e-ait-357ba574-6764496898-9b455" (UID: "4a765418-feb5-41d7-a182-2c22263b21bc") : secret "maas-api-e2e-ait-357ba574-serving-cert" not found
Jul 22 18:43:38.146003 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.145985    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.153851 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.153824    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgc4g\" (UniqueName: \"kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.649186 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.649151    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.651500 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.651477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") pod \"maas-api-e2e-ait-357ba574-6764496898-9b455\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.847393 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.847359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:38.979482 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.979457    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:43:38.981782 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:43:38.981756    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a765418_feb5_41d7_a182_2c22263b21bc.slice/crio-6696a8a6652cceeef01b0f0b3c8ed865a4314b25ebf07e8692962406bc36106f WatchSource:0}: Error finding container 6696a8a6652cceeef01b0f0b3c8ed865a4314b25ebf07e8692962406bc36106f: Status 404 returned error can't find the container with id 6696a8a6652cceeef01b0f0b3c8ed865a4314b25ebf07e8692962406bc36106f
Jul 22 18:43:38.983226 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:38.983209    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:43:39.876019 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:39.875977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" event={"ID":"4a765418-feb5-41d7-a182-2c22263b21bc","Type":"ContainerStarted","Data":"6696a8a6652cceeef01b0f0b3c8ed865a4314b25ebf07e8692962406bc36106f"}
Jul 22 18:43:41.886778 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:41.886741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" event={"ID":"4a765418-feb5-41d7-a182-2c22263b21bc","Type":"ContainerStarted","Data":"1ece9f6edd7c5c92644fae599669c77d11c8d2fc231e4ece35e8c80658138832"}
Jul 22 18:43:41.887190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:41.886843    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:43:41.904457 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:41.904410    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" podStartSLOduration=2.54308547 podStartE2EDuration="4.904394669s" totalImagesPullingTime="2.361309199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:43:41.902880807 +0000 UTC m=+1776.118528210" watchObservedRunningTime="2026-07-22 18:43:41.904394669 +0000 UTC m=+1776.120042072"
Jul 22 18:43:42.769508 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:42.769467    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:43:47.897059 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:43:47.897032    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:44:03.306478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.306401    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"]
Jul 22 18:44:03.313324 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.313304    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:03.315718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.315692    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:44:03.317083 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.317060    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"]
Jul 22 18:44:03.462112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.461908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-357ba574-fp4qt\" (UID: \"8b05acfd-bc3e-4099-9df5-366b738e1517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:03.563330 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.563233    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-357ba574-fp4qt\" (UID: \"8b05acfd-bc3e-4099-9df5-366b738e1517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:03.563915 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.563895    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-357ba574-fp4qt\" (UID: \"8b05acfd-bc3e-4099-9df5-366b738e1517\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:03.624553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.624485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:03.963959 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.963929    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"]
Jul 22 18:44:03.972139 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:44:03.972103    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b05acfd_bc3e_4099_9df5_366b738e1517.slice/crio-7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96 WatchSource:0}: Error finding container 7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96: Status 404 returned error can't find the container with id 7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96
Jul 22 18:44:03.998315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:03.998279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt" event={"ID":"8b05acfd-bc3e-4099-9df5-366b738e1517","Type":"ContainerStarted","Data":"7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96"}
Jul 22 18:44:05.004272 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:05.004182    2580 generic.go:356] "Generic (PLEG): container finished" podID="8b05acfd-bc3e-4099-9df5-366b738e1517" containerID="dc49ecb2fee1ed40a530333707e532d737bedacd5f2d9b13385e7b486d2b1fed" exitCode=0
Jul 22 18:44:05.004272 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:05.004253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt" event={"ID":"8b05acfd-bc3e-4099-9df5-366b738e1517","Type":"ContainerDied","Data":"dc49ecb2fee1ed40a530333707e532d737bedacd5f2d9b13385e7b486d2b1fed"}
Jul 22 18:44:06.142474 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:06.142445    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:06.286237 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:06.286150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca\") pod \"8b05acfd-bc3e-4099-9df5-366b738e1517\" (UID: \"8b05acfd-bc3e-4099-9df5-366b738e1517\") "
Jul 22 18:44:06.286979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:06.286951    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8b05acfd-bc3e-4099-9df5-366b738e1517" (UID: "8b05acfd-bc3e-4099-9df5-366b738e1517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:44:06.387631 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:06.387597    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b05acfd-bc3e-4099-9df5-366b738e1517-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:07.014935 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:07.014905    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"
Jul 22 18:44:07.015112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:07.014905    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt" event={"ID":"8b05acfd-bc3e-4099-9df5-366b738e1517","Type":"ContainerDied","Data":"7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96"}
Jul 22 18:44:07.015112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:07.015026    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7666802e917dcaddf5571f8f4048180c5d15b407fc33d98ec6132e699c005c96"
Jul 22 18:44:13.932899 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:13.932866    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:44:13.933350 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:13.933090    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" podUID="4a765418-feb5-41d7-a182-2c22263b21bc" containerName="maas-api" containerID="cri-o://1ece9f6edd7c5c92644fae599669c77d11c8d2fc231e4ece35e8c80658138832" gracePeriod=30
Jul 22 18:44:14.049047 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.049009    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a765418-feb5-41d7-a182-2c22263b21bc" containerID="1ece9f6edd7c5c92644fae599669c77d11c8d2fc231e4ece35e8c80658138832" exitCode=0
Jul 22 18:44:14.049189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.049069    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" event={"ID":"4a765418-feb5-41d7-a182-2c22263b21bc","Type":"ContainerDied","Data":"1ece9f6edd7c5c92644fae599669c77d11c8d2fc231e4ece35e8c80658138832"}
Jul 22 18:44:14.178161 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.178135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:44:14.247791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.247711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca\") pod \"4a765418-feb5-41d7-a182-2c22263b21bc\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") "
Jul 22 18:44:14.247791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.247762    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") pod \"4a765418-feb5-41d7-a182-2c22263b21bc\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") "
Jul 22 18:44:14.248062 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.247828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgc4g\" (UniqueName: \"kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g\") pod \"4a765418-feb5-41d7-a182-2c22263b21bc\" (UID: \"4a765418-feb5-41d7-a182-2c22263b21bc\") "
Jul 22 18:44:14.248147 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.248121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a765418-feb5-41d7-a182-2c22263b21bc" (UID: "4a765418-feb5-41d7-a182-2c22263b21bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:44:14.249982 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.249952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g" (OuterVolumeSpecName: "kube-api-access-wgc4g") pod "4a765418-feb5-41d7-a182-2c22263b21bc" (UID: "4a765418-feb5-41d7-a182-2c22263b21bc"). InnerVolumeSpecName "kube-api-access-wgc4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:44:14.250097 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.250014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a765418-feb5-41d7-a182-2c22263b21bc" (UID: "4a765418-feb5-41d7-a182-2c22263b21bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:44:14.348996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.348966    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgc4g\" (UniqueName: \"kubernetes.io/projected/4a765418-feb5-41d7-a182-2c22263b21bc-kube-api-access-wgc4g\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:14.348996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.348990    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a765418-feb5-41d7-a182-2c22263b21bc-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:14.348996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:14.349000    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a765418-feb5-41d7-a182-2c22263b21bc-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:15.055197 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:15.055156    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455" event={"ID":"4a765418-feb5-41d7-a182-2c22263b21bc","Type":"ContainerDied","Data":"6696a8a6652cceeef01b0f0b3c8ed865a4314b25ebf07e8692962406bc36106f"}
Jul 22 18:44:15.055197 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:15.055184    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"
Jul 22 18:44:15.055735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:15.055233    2580 scope.go:117] "RemoveContainer" containerID="1ece9f6edd7c5c92644fae599669c77d11c8d2fc231e4ece35e8c80658138832"
Jul 22 18:44:15.073031 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:15.073000    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:44:15.079770 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:15.079748    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-357ba574-6764496898-9b455"]
Jul 22 18:44:16.320315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:16.320280    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a765418-feb5-41d7-a182-2c22263b21bc" path="/var/lib/kubelet/pods/4a765418-feb5-41d7-a182-2c22263b21bc/volumes"
Jul 22 18:44:16.973313 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:16.973281    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:44:18.473141 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:18.473105    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"]
Jul 22 18:44:18.480531 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:18.477002    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-357ba574-fp4qt"]
Jul 22 18:44:20.319958 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:20.319923    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b05acfd-bc3e-4099-9df5-366b738e1517" path="/var/lib/kubelet/pods/8b05acfd-bc3e-4099-9df5-366b738e1517/volumes"
Jul 22 18:44:22.153046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153017    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:22.153404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153360    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b05acfd-bc3e-4099-9df5-366b738e1517" containerName="revoke-keys"
Jul 22 18:44:22.153404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153372    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b05acfd-bc3e-4099-9df5-366b738e1517" containerName="revoke-keys"
Jul 22 18:44:22.153404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153389    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a765418-feb5-41d7-a182-2c22263b21bc" containerName="maas-api"
Jul 22 18:44:22.153404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153395    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a765418-feb5-41d7-a182-2c22263b21bc" containerName="maas-api"
Jul 22 18:44:22.153533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153449    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b05acfd-bc3e-4099-9df5-366b738e1517" containerName="revoke-keys"
Jul 22 18:44:22.153533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.153458    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a765418-feb5-41d7-a182-2c22263b21bc" containerName="maas-api"
Jul 22 18:44:22.156661 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.156640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.159032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.159014    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-e37eed10-serving-cert\""
Jul 22 18:44:22.159125 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.159055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:44:22.167386 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.167358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:22.208522 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.208492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.208624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.208525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.208624 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.208551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj229\" (UniqueName: \"kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.309745 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.309721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.309925 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.309750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.309925 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.309776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj229\" (UniqueName: \"kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.309925 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:44:22.309884    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-serving-cert: secret "maas-api-e2e-migrate-e37eed10-serving-cert" not found
Jul 22 18:44:22.310037 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:44:22.309963    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls podName:285b46f8-5d27-4307-811f-4b68b5ad55b2 nodeName:}" failed. No retries permitted until 2026-07-22 18:44:22.809942415 +0000 UTC m=+1817.025589796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls") pod "maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" (UID: "285b46f8-5d27-4307-811f-4b68b5ad55b2") : secret "maas-api-e2e-migrate-e37eed10-serving-cert" not found
Jul 22 18:44:22.310353 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.310336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.320845 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.320796    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj229\" (UniqueName: \"kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.813409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.813379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:22.815705 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:22.815680    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") pod \"maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:23.067126 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:23.067044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:23.200680 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:23.200658    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:23.203318 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:44:23.203292    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285b46f8_5d27_4307_811f_4b68b5ad55b2.slice/crio-87dfde1af9d1080e7a7b8dd2cd6e76532e973d523ba5040ec1767a824cff0490 WatchSource:0}: Error finding container 87dfde1af9d1080e7a7b8dd2cd6e76532e973d523ba5040ec1767a824cff0490: Status 404 returned error can't find the container with id 87dfde1af9d1080e7a7b8dd2cd6e76532e973d523ba5040ec1767a824cff0490
Jul 22 18:44:24.100300 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:24.100262    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" event={"ID":"285b46f8-5d27-4307-811f-4b68b5ad55b2","Type":"ContainerStarted","Data":"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"}
Jul 22 18:44:24.100523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:24.100320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" event={"ID":"285b46f8-5d27-4307-811f-4b68b5ad55b2","Type":"ContainerStarted","Data":"87dfde1af9d1080e7a7b8dd2cd6e76532e973d523ba5040ec1767a824cff0490"}
Jul 22 18:44:24.100523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:24.100342    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:24.117364 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:24.117280    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" podStartSLOduration=1.454777964 podStartE2EDuration="2.11726544s" totalImagesPullingTime="662.487476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:24.115647236 +0000 UTC m=+1818.331294639" watchObservedRunningTime="2026-07-22 18:44:24.11726544 +0000 UTC m=+1818.332912843"
Jul 22 18:44:26.880341 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:26.880305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:44:30.110594 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:30.110568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:47.101955 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.101916    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"]
Jul 22 18:44:47.105241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.105223    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:47.107446 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.107427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:44:47.112981 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.112924    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"]
Jul 22 18:44:47.212306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.212272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z\" (UID: \"7fc330b3-83ef-40ca-b328-51547cd24ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:47.312990 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.312952    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z\" (UID: \"7fc330b3-83ef-40ca-b328-51547cd24ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:47.313567 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.313546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z\" (UID: \"7fc330b3-83ef-40ca-b328-51547cd24ef1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:47.417481 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.417403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:47.545343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:47.545305    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"]
Jul 22 18:44:47.550922 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:44:47.550894    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc330b3_83ef_40ca_b328_51547cd24ef1.slice/crio-1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55 WatchSource:0}: Error finding container 1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55: Status 404 returned error can't find the container with id 1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55
Jul 22 18:44:48.216410 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:48.216372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z" event={"ID":"7fc330b3-83ef-40ca-b328-51547cd24ef1","Type":"ContainerStarted","Data":"f52773d5eea728df85abea37716d217ae0835ef69a21a2a62a00376928e50d1a"}
Jul 22 18:44:48.216827 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:48.216452    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z" event={"ID":"7fc330b3-83ef-40ca-b328-51547cd24ef1","Type":"ContainerStarted","Data":"1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55"}
Jul 22 18:44:48.233791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:48.233739    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z" podStartSLOduration=1.233720284 podStartE2EDuration="1.233720284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:44:48.233615683 +0000 UTC m=+1842.449263087" watchObservedRunningTime="2026-07-22 18:44:48.233720284 +0000 UTC m=+1842.449367688"
Jul 22 18:44:49.222684 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:49.222649    2580 generic.go:356] "Generic (PLEG): container finished" podID="7fc330b3-83ef-40ca-b328-51547cd24ef1" containerID="f52773d5eea728df85abea37716d217ae0835ef69a21a2a62a00376928e50d1a" exitCode=0
Jul 22 18:44:49.223152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:49.222734    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z" event={"ID":"7fc330b3-83ef-40ca-b328-51547cd24ef1","Type":"ContainerDied","Data":"f52773d5eea728df85abea37716d217ae0835ef69a21a2a62a00376928e50d1a"}
Jul 22 18:44:50.359723 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:50.359701    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:50.439704 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:50.439675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca\") pod \"7fc330b3-83ef-40ca-b328-51547cd24ef1\" (UID: \"7fc330b3-83ef-40ca-b328-51547cd24ef1\") "
Jul 22 18:44:50.440019 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:50.439999    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fc330b3-83ef-40ca-b328-51547cd24ef1" (UID: "7fc330b3-83ef-40ca-b328-51547cd24ef1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:44:50.541109 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:50.541022    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc330b3-83ef-40ca-b328-51547cd24ef1-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:51.233458 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:51.233431    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"
Jul 22 18:44:51.233712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:51.233478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z" event={"ID":"7fc330b3-83ef-40ca-b328-51547cd24ef1","Type":"ContainerDied","Data":"1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55"}
Jul 22 18:44:51.233712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:51.233522    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa6912c08928ae8eee72202f4052f9f80c28d31e43b0c7fb54d370dcfad4f55"
Jul 22 18:44:57.643610 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:57.643574    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:57.644079 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:57.643787    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" podUID="285b46f8-5d27-4307-811f-4b68b5ad55b2" containerName="maas-api" containerID="cri-o://003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4" gracePeriod=30
Jul 22 18:44:57.892707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:57.892683    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:58.010407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.010375    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca\") pod \"285b46f8-5d27-4307-811f-4b68b5ad55b2\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") "
Jul 22 18:44:58.010585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.010476    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") pod \"285b46f8-5d27-4307-811f-4b68b5ad55b2\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") "
Jul 22 18:44:58.010585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.010506    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bj229\" (UniqueName: \"kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229\") pod \"285b46f8-5d27-4307-811f-4b68b5ad55b2\" (UID: \"285b46f8-5d27-4307-811f-4b68b5ad55b2\") "
Jul 22 18:44:58.010729 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.010706    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "285b46f8-5d27-4307-811f-4b68b5ad55b2" (UID: "285b46f8-5d27-4307-811f-4b68b5ad55b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:44:58.012664 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.012638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "285b46f8-5d27-4307-811f-4b68b5ad55b2" (UID: "285b46f8-5d27-4307-811f-4b68b5ad55b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:44:58.012664 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.012646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229" (OuterVolumeSpecName: "kube-api-access-bj229") pod "285b46f8-5d27-4307-811f-4b68b5ad55b2" (UID: "285b46f8-5d27-4307-811f-4b68b5ad55b2"). InnerVolumeSpecName "kube-api-access-bj229". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:44:58.111771 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.111737    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/285b46f8-5d27-4307-811f-4b68b5ad55b2-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:58.111771 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.111766    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj229\" (UniqueName: \"kubernetes.io/projected/285b46f8-5d27-4307-811f-4b68b5ad55b2-kube-api-access-bj229\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:58.111771 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.111777    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/285b46f8-5d27-4307-811f-4b68b5ad55b2-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:44:58.270182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.270090    2580 generic.go:356] "Generic (PLEG): container finished" podID="285b46f8-5d27-4307-811f-4b68b5ad55b2" containerID="003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4" exitCode=0
Jul 22 18:44:58.270182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.270133    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"
Jul 22 18:44:58.270182 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.270140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" event={"ID":"285b46f8-5d27-4307-811f-4b68b5ad55b2","Type":"ContainerDied","Data":"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"}
Jul 22 18:44:58.270441 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.270189    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv" event={"ID":"285b46f8-5d27-4307-811f-4b68b5ad55b2","Type":"ContainerDied","Data":"87dfde1af9d1080e7a7b8dd2cd6e76532e973d523ba5040ec1767a824cff0490"}
Jul 22 18:44:58.270441 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.270212    2580 scope.go:117] "RemoveContainer" containerID="003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"
Jul 22 18:44:58.279244 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.279223    2580 scope.go:117] "RemoveContainer" containerID="003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"
Jul 22 18:44:58.279533 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:44:58.279509    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4\": container with ID starting with 003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4 not found: ID does not exist" containerID="003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"
Jul 22 18:44:58.279598 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.279542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4"} err="failed to get container status \"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4\": rpc error: code = NotFound desc = could not find container \"003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4\": container with ID starting with 003a52f92ef1534caa868e3bf8b169cb5325eb0def38e7f686ced6aa1dda90c4 not found: ID does not exist"
Jul 22 18:44:58.295264 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.295237    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:58.300718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.300697    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-e37eed10-7d49db747c-mpwfv"]
Jul 22 18:44:58.321137 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:44:58.321110    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="285b46f8-5d27-4307-811f-4b68b5ad55b2" path="/var/lib/kubelet/pods/285b46f8-5d27-4307-811f-4b68b5ad55b2/volumes"
Jul 22 18:45:00.133455 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.133413    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"]
Jul 22 18:45:00.133972 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.133953    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285b46f8-5d27-4307-811f-4b68b5ad55b2" containerName="maas-api"
Jul 22 18:45:00.134064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.133975    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="285b46f8-5d27-4307-811f-4b68b5ad55b2" containerName="maas-api"
Jul 22 18:45:00.134064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.134006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc330b3-83ef-40ca-b328-51547cd24ef1" containerName="revoke-keys"
Jul 22 18:45:00.134064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.134015    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc330b3-83ef-40ca-b328-51547cd24ef1" containerName="revoke-keys"
Jul 22 18:45:00.134215 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.134104    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc330b3-83ef-40ca-b328-51547cd24ef1" containerName="revoke-keys"
Jul 22 18:45:00.134215 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.134118    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="285b46f8-5d27-4307-811f-4b68b5ad55b2" containerName="maas-api"
Jul 22 18:45:00.138332 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.138312    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"
Jul 22 18:45:00.140935 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.140861    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:45:00.141346 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.141157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"]
Jul 22 18:45:00.149889 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.149872    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"
Jul 22 18:45:00.279173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.278981    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"]
Jul 22 18:45:00.281802 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:45:00.281774    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb905a0_bc69_490a_9445_5528e516b493.slice/crio-3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639 WatchSource:0}: Error finding container 3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639: Status 404 returned error can't find the container with id 3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639
Jul 22 18:45:00.566120 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:00.566085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:45:01.287880 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:01.287846    2580 generic.go:356] "Generic (PLEG): container finished" podID="0fb905a0-bc69-490a-9445-5528e516b493" containerID="023263309a733b194be401404e3eec74a83cc83ae8e6eafebf91dd41ce74b699" exitCode=0
Jul 22 18:45:01.288298 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:01.287936    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j" event={"ID":"0fb905a0-bc69-490a-9445-5528e516b493","Type":"ContainerDied","Data":"023263309a733b194be401404e3eec74a83cc83ae8e6eafebf91dd41ce74b699"}
Jul 22 18:45:01.288298 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:01.287984    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j" event={"ID":"0fb905a0-bc69-490a-9445-5528e516b493","Type":"ContainerStarted","Data":"3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639"}
Jul 22 18:45:02.180484 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:02.180448    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"]
Jul 22 18:45:02.189686 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:02.189657    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e37eed10-cww9z"]
Jul 22 18:45:02.324310 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:02.324270    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc330b3-83ef-40ca-b328-51547cd24ef1" path="/var/lib/kubelet/pods/7fc330b3-83ef-40ca-b328-51547cd24ef1/volumes"
Jul 22 18:45:02.432298 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:02.432238    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"
Jul 22 18:45:03.298593 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:03.298565    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j"
Jul 22 18:45:03.298750 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:03.298564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745765-6l79j" event={"ID":"0fb905a0-bc69-490a-9445-5528e516b493","Type":"ContainerDied","Data":"3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639"}
Jul 22 18:45:03.298750 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:03.298701    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c905fd55fbc01aa7d355164be9c0fe795bf8c102e9c4a26c29f81285e8ec639"
Jul 22 18:45:05.330175 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.330143    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:05.330539 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.330491    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb905a0-bc69-490a-9445-5528e516b493" containerName="cleanup"
Jul 22 18:45:05.330539 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.330502    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb905a0-bc69-490a-9445-5528e516b493" containerName="cleanup"
Jul 22 18:45:05.330611 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.330567    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb905a0-bc69-490a-9445-5528e516b493" containerName="cleanup"
Jul 22 18:45:05.333445 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.333427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.336252 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.336227    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:45:05.336510 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.336494    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-cc99fc0b-serving-cert\""
Jul 22 18:45:05.356300 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.356263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:05.475496 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.475463    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.475651 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.475517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.475651 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.475542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w68x\" (UniqueName: \"kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.577021 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.576980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.577190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.577027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w68x\" (UniqueName: \"kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.577190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.577121    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.577190 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:05.577137    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-serving-cert: secret "maas-api-e2e-ait-cc99fc0b-serving-cert" not found
Jul 22 18:45:05.577324 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:05.577206    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls podName:6ab2e459-9994-4697-a6fb-c1f188dd24cb nodeName:}" failed. No retries permitted until 2026-07-22 18:45:06.077187251 +0000 UTC m=+1860.292834639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls") pod "maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" (UID: "6ab2e459-9994-4697-a6fb-c1f188dd24cb") : secret "maas-api-e2e-ait-cc99fc0b-serving-cert" not found
Jul 22 18:45:05.577833 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.577790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:05.586235 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:05.586179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w68x\" (UniqueName: \"kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:06.081094 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:06.081058    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:06.083493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:06.083474    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") pod \"maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:06.246067 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:06.246031    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:45:06.254040 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:06.254011    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:06.392670 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:06.392648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:06.394978 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:45:06.394952    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab2e459_9994_4697_a6fb_c1f188dd24cb.slice/crio-5645ec168e5be0bb022bec49bbae7bb87ad75e4b3da644a0caf6632598e5f05e WatchSource:0}: Error finding container 5645ec168e5be0bb022bec49bbae7bb87ad75e4b3da644a0caf6632598e5f05e: Status 404 returned error can't find the container with id 5645ec168e5be0bb022bec49bbae7bb87ad75e4b3da644a0caf6632598e5f05e
Jul 22 18:45:07.323264 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:07.323224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" event={"ID":"6ab2e459-9994-4697-a6fb-c1f188dd24cb","Type":"ContainerStarted","Data":"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"}
Jul 22 18:45:07.323469 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:07.323318    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:07.323469 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:07.323337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" event={"ID":"6ab2e459-9994-4697-a6fb-c1f188dd24cb","Type":"ContainerStarted","Data":"5645ec168e5be0bb022bec49bbae7bb87ad75e4b3da644a0caf6632598e5f05e"}
Jul 22 18:45:07.340014 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:07.339953    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" podStartSLOduration=1.928436981 podStartE2EDuration="2.339937326s" totalImagesPullingTime="411.500345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:07.33830539 +0000 UTC m=+1861.553952792" watchObservedRunningTime="2026-07-22 18:45:07.339937326 +0000 UTC m=+1861.555584729"
Jul 22 18:45:11.168611 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:11.168574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:45:13.333940 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:13.333907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:31.240228 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.240149    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"]
Jul 22 18:45:31.243645 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.243630    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:31.246219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.246196    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:45:31.254038 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.252865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"]
Jul 22 18:45:31.280758 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.280732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5\" (UID: \"dd8ef6fd-0a0a-498c-b72b-749e50d29c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:31.382014 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.381988    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5\" (UID: \"dd8ef6fd-0a0a-498c-b72b-749e50d29c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:31.382513 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.382496    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5\" (UID: \"dd8ef6fd-0a0a-498c-b72b-749e50d29c32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:31.563518 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.563449    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:31.704909 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:31.704871    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"]
Jul 22 18:45:31.707283 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:45:31.707251    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8ef6fd_0a0a_498c_b72b_749e50d29c32.slice/crio-b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa WatchSource:0}: Error finding container b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa: Status 404 returned error can't find the container with id b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa
Jul 22 18:45:32.440897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:32.440864    2580 generic.go:356] "Generic (PLEG): container finished" podID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" containerID="f1a59e27e091a7bc57a14dd1dd4a64697afa8358ef4fa476caf9edaf09c20599" exitCode=0
Jul 22 18:45:32.441280 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:32.440925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5" event={"ID":"dd8ef6fd-0a0a-498c-b72b-749e50d29c32","Type":"ContainerDied","Data":"f1a59e27e091a7bc57a14dd1dd4a64697afa8358ef4fa476caf9edaf09c20599"}
Jul 22 18:45:32.441280 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:32.440971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5" event={"ID":"dd8ef6fd-0a0a-498c-b72b-749e50d29c32","Type":"ContainerStarted","Data":"b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa"}
Jul 22 18:45:33.579735 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:33.579713    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:33.600439 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:33.600412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca\") pod \"dd8ef6fd-0a0a-498c-b72b-749e50d29c32\" (UID: \"dd8ef6fd-0a0a-498c-b72b-749e50d29c32\") "
Jul 22 18:45:33.600724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:33.600701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd8ef6fd-0a0a-498c-b72b-749e50d29c32" (UID: "dd8ef6fd-0a0a-498c-b72b-749e50d29c32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:45:33.701480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:33.701409    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8ef6fd-0a0a-498c-b72b-749e50d29c32-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:45:34.451001 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:34.450963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5" event={"ID":"dd8ef6fd-0a0a-498c-b72b-749e50d29c32","Type":"ContainerDied","Data":"b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa"}
Jul 22 18:45:34.451001 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:34.450997    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:34.451184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:34.451018    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03aa6f33541a8e75efd4c6b789cc79a106d18be5219ca6c8d87a59730f4f8aa"
Jul 22 18:45:41.772604 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:41.772570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:41.773100 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:41.772761    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" podUID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" containerName="maas-api" containerID="cri-o://a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4" gracePeriod=30
Jul 22 18:45:42.007603 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.007570    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:42.071158 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.071077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w68x\" (UniqueName: \"kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x\") pod \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") "
Jul 22 18:45:42.071305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.071181    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca\") pod \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") "
Jul 22 18:45:42.071305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.071271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") pod \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\" (UID: \"6ab2e459-9994-4697-a6fb-c1f188dd24cb\") "
Jul 22 18:45:42.071500 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.071472    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ab2e459-9994-4697-a6fb-c1f188dd24cb" (UID: "6ab2e459-9994-4697-a6fb-c1f188dd24cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:45:42.073289 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.073262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ab2e459-9994-4697-a6fb-c1f188dd24cb" (UID: "6ab2e459-9994-4697-a6fb-c1f188dd24cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:45:42.073522 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.073500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x" (OuterVolumeSpecName: "kube-api-access-7w68x") pod "6ab2e459-9994-4697-a6fb-c1f188dd24cb" (UID: "6ab2e459-9994-4697-a6fb-c1f188dd24cb"). InnerVolumeSpecName "kube-api-access-7w68x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:45:42.172895 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.172859    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ab2e459-9994-4697-a6fb-c1f188dd24cb-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:45:42.172895 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.172892    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ab2e459-9994-4697-a6fb-c1f188dd24cb-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:45:42.172895 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.172902    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w68x\" (UniqueName: \"kubernetes.io/projected/6ab2e459-9994-4697-a6fb-c1f188dd24cb-kube-api-access-7w68x\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:45:42.490119 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.490085    2580 generic.go:356] "Generic (PLEG): container finished" podID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" containerID="a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4" exitCode=0
Jul 22 18:45:42.490265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.490144    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"
Jul 22 18:45:42.490265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.490165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" event={"ID":"6ab2e459-9994-4697-a6fb-c1f188dd24cb","Type":"ContainerDied","Data":"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"}
Jul 22 18:45:42.490265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.490216    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm" event={"ID":"6ab2e459-9994-4697-a6fb-c1f188dd24cb","Type":"ContainerDied","Data":"5645ec168e5be0bb022bec49bbae7bb87ad75e4b3da644a0caf6632598e5f05e"}
Jul 22 18:45:42.490265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.490238    2580 scope.go:117] "RemoveContainer" containerID="a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"
Jul 22 18:45:42.498681 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.498663    2580 scope.go:117] "RemoveContainer" containerID="a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"
Jul 22 18:45:42.498937 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:42.498917    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4\": container with ID starting with a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4 not found: ID does not exist" containerID="a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"
Jul 22 18:45:42.499006 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.498947    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4"} err="failed to get container status \"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4\": rpc error: code = NotFound desc = could not find container \"a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4\": container with ID starting with a9dddbb3111afa13320d18ffa54dedd779c8b306c1d353320997319d6ebab5e4 not found: ID does not exist"
Jul 22 18:45:42.506752 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.506729    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:42.510331 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:42.510311    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-cc99fc0b-5d585fb49f-ff6cm"]
Jul 22 18:45:44.320938 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:44.320905    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" path="/var/lib/kubelet/pods/6ab2e459-9994-4697-a6fb-c1f188dd24cb/volumes"
Jul 22 18:45:44.564835 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:44.564789    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:45:46.314628 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:46.314593    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"]
Jul 22 18:45:46.318917 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:46.318888    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"
Jul 22 18:45:46.320408 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:46.320389    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-cc99fc0b-pn7b5"]
Jul 22 18:45:48.320063 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:48.320032    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" path="/var/lib/kubelet/pods/dd8ef6fd-0a0a-498c-b72b-749e50d29c32/volumes"
Jul 22 18:45:50.512385 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.512354    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:45:50.512907 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.512888    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" containerName="maas-api"
Jul 22 18:45:50.513000 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.512909    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" containerName="maas-api"
Jul 22 18:45:50.513000 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.512932    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" containerName="revoke-keys"
Jul 22 18:45:50.513000 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.512941    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" containerName="revoke-keys"
Jul 22 18:45:50.513158 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.513040    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab2e459-9994-4697-a6fb-c1f188dd24cb" containerName="maas-api"
Jul 22 18:45:50.513158 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.513052    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8ef6fd-0a0a-498c-b72b-749e50d29c32" containerName="revoke-keys"
Jul 22 18:45:50.516207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.516184    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.518714 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.518686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:45:50.518850 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.518685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-95d099d7-serving-cert\""
Jul 22 18:45:50.525507 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.525485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:45:50.644576 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.644536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.644576 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.644577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4wx\" (UniqueName: \"kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.644797 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.644604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.745431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.745391    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.745431 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.745435    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4wx\" (UniqueName: \"kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.745689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.745468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.745689 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:50.745592    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-serving-cert: secret "maas-api-e2e-derive-95d099d7-serving-cert" not found
Jul 22 18:45:50.745689 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:45:50.745670    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls podName:fb06aa22-b384-47ab-8a5a-b53efa844c2f nodeName:}" failed. No retries permitted until 2026-07-22 18:45:51.245648932 +0000 UTC m=+1905.461296312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls") pod "maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" (UID: "fb06aa22-b384-47ab-8a5a-b53efa844c2f") : secret "maas-api-e2e-derive-95d099d7-serving-cert" not found
Jul 22 18:45:50.746063 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.746043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:50.756290 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:50.756269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4wx\" (UniqueName: \"kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:51.249799 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:51.249764    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:51.252319 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:51.252287    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") pod \"maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:51.428150 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:51.428117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:51.567157 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:51.567130    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:45:51.568962 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:45:51.568935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb06aa22_b384_47ab_8a5a_b53efa844c2f.slice/crio-eb62ce3a2d77b9409fc6707a953fb3db8a17f92afc9bc75dfe337fd41c144a5f WatchSource:0}: Error finding container eb62ce3a2d77b9409fc6707a953fb3db8a17f92afc9bc75dfe337fd41c144a5f: Status 404 returned error can't find the container with id eb62ce3a2d77b9409fc6707a953fb3db8a17f92afc9bc75dfe337fd41c144a5f
Jul 22 18:45:52.544001 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:52.543968    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" event={"ID":"fb06aa22-b384-47ab-8a5a-b53efa844c2f","Type":"ContainerStarted","Data":"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"}
Jul 22 18:45:52.544189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:52.544020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" event={"ID":"fb06aa22-b384-47ab-8a5a-b53efa844c2f","Type":"ContainerStarted","Data":"eb62ce3a2d77b9409fc6707a953fb3db8a17f92afc9bc75dfe337fd41c144a5f"}
Jul 22 18:45:52.544189 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:52.544147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:45:52.561460 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:52.561411    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" podStartSLOduration=2.006769214 podStartE2EDuration="2.561397304s" totalImagesPullingTime="554.62809ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:45:52.559156892 +0000 UTC m=+1906.774804296" watchObservedRunningTime="2026-07-22 18:45:52.561397304 +0000 UTC m=+1906.777044769"
Jul 22 18:45:55.382751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:55.382713    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:45:58.555250 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:45:58.555211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:46:15.392465 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.392435    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"]
Jul 22 18:46:15.395852 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.395836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:15.398273 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.398251    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:46:15.402158 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.402137    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"]
Jul 22 18:46:15.552977 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.552951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6\" (UID: \"50f222e2-9f93-407f-a0ea-6a335b8370e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:15.654226 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.654165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6\" (UID: \"50f222e2-9f93-407f-a0ea-6a335b8370e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:15.654718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.654699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6\" (UID: \"50f222e2-9f93-407f-a0ea-6a335b8370e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:15.708293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.708271    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:15.831483 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:15.831458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"]
Jul 22 18:46:15.833305 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:46:15.833281    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f222e2_9f93_407f_a0ea_6a335b8370e2.slice/crio-74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1 WatchSource:0}: Error finding container 74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1: Status 404 returned error can't find the container with id 74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1
Jul 22 18:46:16.654868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:16.654839    2580 generic.go:356] "Generic (PLEG): container finished" podID="50f222e2-9f93-407f-a0ea-6a335b8370e2" containerID="afd1227457880e180280352c27c1168caedadf07d1bb81c8279d4bd3736bc5db" exitCode=0
Jul 22 18:46:16.655191 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:16.654914    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6" event={"ID":"50f222e2-9f93-407f-a0ea-6a335b8370e2","Type":"ContainerDied","Data":"afd1227457880e180280352c27c1168caedadf07d1bb81c8279d4bd3736bc5db"}
Jul 22 18:46:16.655191 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:16.654958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6" event={"ID":"50f222e2-9f93-407f-a0ea-6a335b8370e2","Type":"ContainerStarted","Data":"74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1"}
Jul 22 18:46:17.785241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:17.785215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:17.873099 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:17.873074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca\") pod \"50f222e2-9f93-407f-a0ea-6a335b8370e2\" (UID: \"50f222e2-9f93-407f-a0ea-6a335b8370e2\") "
Jul 22 18:46:17.873404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:17.873385    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50f222e2-9f93-407f-a0ea-6a335b8370e2" (UID: "50f222e2-9f93-407f-a0ea-6a335b8370e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:46:17.974535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:17.974508    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50f222e2-9f93-407f-a0ea-6a335b8370e2-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:46:18.665763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:18.665685    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"
Jul 22 18:46:18.665926 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:18.665689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6" event={"ID":"50f222e2-9f93-407f-a0ea-6a335b8370e2","Type":"ContainerDied","Data":"74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1"}
Jul 22 18:46:18.665926 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:18.665834    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c63a1eb25b584e77af9a0b150d8df2f66fe6d2a153a6e2c83ab93ce54959b1"
Jul 22 18:46:25.945200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:25.945166    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:46:25.945642 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:25.945454    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" podUID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" containerName="maas-api" containerID="cri-o://b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad" gracePeriod=30
Jul 22 18:46:26.191705 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.191684    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:46:26.336594 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.336518    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") pod \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") "
Jul 22 18:46:26.336594 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.336549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s4wx\" (UniqueName: \"kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx\") pod \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") "
Jul 22 18:46:26.336799 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.336598    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca\") pod \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\" (UID: \"fb06aa22-b384-47ab-8a5a-b53efa844c2f\") "
Jul 22 18:46:26.337051 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.337028    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb06aa22-b384-47ab-8a5a-b53efa844c2f" (UID: "fb06aa22-b384-47ab-8a5a-b53efa844c2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:46:26.338447 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.338424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb06aa22-b384-47ab-8a5a-b53efa844c2f" (UID: "fb06aa22-b384-47ab-8a5a-b53efa844c2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:46:26.338662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.338638    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx" (OuterVolumeSpecName: "kube-api-access-9s4wx") pod "fb06aa22-b384-47ab-8a5a-b53efa844c2f" (UID: "fb06aa22-b384-47ab-8a5a-b53efa844c2f"). InnerVolumeSpecName "kube-api-access-9s4wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:46:26.437317 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.437286    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb06aa22-b384-47ab-8a5a-b53efa844c2f-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:46:26.437317 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.437317    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb06aa22-b384-47ab-8a5a-b53efa844c2f-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:46:26.437474 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.437333    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4wx\" (UniqueName: \"kubernetes.io/projected/fb06aa22-b384-47ab-8a5a-b53efa844c2f-kube-api-access-9s4wx\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:46:26.706208 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.706169    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" containerID="b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad" exitCode=0
Jul 22 18:46:26.706361 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.706230    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"
Jul 22 18:46:26.706361 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.706261    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" event={"ID":"fb06aa22-b384-47ab-8a5a-b53efa844c2f","Type":"ContainerDied","Data":"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"}
Jul 22 18:46:26.706361 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.706339    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj" event={"ID":"fb06aa22-b384-47ab-8a5a-b53efa844c2f","Type":"ContainerDied","Data":"eb62ce3a2d77b9409fc6707a953fb3db8a17f92afc9bc75dfe337fd41c144a5f"}
Jul 22 18:46:26.706521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.706362    2580 scope.go:117] "RemoveContainer" containerID="b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"
Jul 22 18:46:26.715603 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.715571    2580 scope.go:117] "RemoveContainer" containerID="b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"
Jul 22 18:46:26.715833 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:46:26.715794    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad\": container with ID starting with b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad not found: ID does not exist" containerID="b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"
Jul 22 18:46:26.715911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.715845    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad"} err="failed to get container status \"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad\": rpc error: code = NotFound desc = could not find container \"b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad\": container with ID starting with b77c08c08d74113f8f5ae4292b2cf89ea7cb7791d151b6fe918cafb844fd84ad not found: ID does not exist"
Jul 22 18:46:26.727328 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.727300    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:46:26.729133 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:26.729110    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95d099d7-74f7545cd-5lcvj"]
Jul 22 18:46:28.320591 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:28.320562    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" path="/var/lib/kubelet/pods/fb06aa22-b384-47ab-8a5a-b53efa844c2f/volumes"
Jul 22 18:46:28.973108 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:28.973077    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:46:30.466429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:30.466402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"]
Jul 22 18:46:30.468534 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:30.468509    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95d099d7-r2gh6"]
Jul 22 18:46:32.320497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:46:32.320464    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f222e2-9f93-407f-a0ea-6a335b8370e2" path="/var/lib/kubelet/pods/50f222e2-9f93-407f-a0ea-6a335b8370e2/volumes"
Jul 22 18:47:56.007200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:47:56.007162    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:48:05.772631 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:05.772599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:48:11.946205 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946173    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:11.946585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946556    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f222e2-9f93-407f-a0ea-6a335b8370e2" containerName="revoke-keys"
Jul 22 18:48:11.946585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946568    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f222e2-9f93-407f-a0ea-6a335b8370e2" containerName="revoke-keys"
Jul 22 18:48:11.946585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946580    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" containerName="maas-api"
Jul 22 18:48:11.946585 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946586    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" containerName="maas-api"
Jul 22 18:48:11.946749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946647    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb06aa22-b384-47ab-8a5a-b53efa844c2f" containerName="maas-api"
Jul 22 18:48:11.946749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.946656    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f222e2-9f93-407f-a0ea-6a335b8370e2" containerName="revoke-keys"
Jul 22 18:48:11.949458 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.949437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:11.951689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.951668    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d85e8a51-serving-cert\""
Jul 22 18:48:11.951826 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.951791    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:48:11.958559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:11.958537    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:12.088972 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.088945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.089143 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.088998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.089143 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.089023    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.189883 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.189841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.190070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.189986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.190070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.190046    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.190584 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.190562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.192163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.192144    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.201137 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.201078    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp\") pod \"maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.260821 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.260781    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:12.401257 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:12.401231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:12.402761 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:48:12.402725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2acbf4d_7570_41ab_8343_ad244777f543.slice/crio-c2de36ba76ca0f63ec850f91344a0b026d8c393592533d7883c9907d022b1cd7 WatchSource:0}: Error finding container c2de36ba76ca0f63ec850f91344a0b026d8c393592533d7883c9907d022b1cd7: Status 404 returned error can't find the container with id c2de36ba76ca0f63ec850f91344a0b026d8c393592533d7883c9907d022b1cd7
Jul 22 18:48:13.209544 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:13.209509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" event={"ID":"c2acbf4d-7570-41ab-8343-ad244777f543","Type":"ContainerStarted","Data":"8ecbf668a9f8db40469e36ec7110a3ff2f9ddf9aafb525484051f2aad53454d1"}
Jul 22 18:48:13.209927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:13.209589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" event={"ID":"c2acbf4d-7570-41ab-8343-ad244777f543","Type":"ContainerStarted","Data":"c2de36ba76ca0f63ec850f91344a0b026d8c393592533d7883c9907d022b1cd7"}
Jul 22 18:48:13.209927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:13.209641    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:13.227821 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:13.227772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" podStartSLOduration=1.809756734 podStartE2EDuration="2.227758981s" totalImagesPullingTime="418.002247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:48:13.227046139 +0000 UTC m=+2047.442693568" watchObservedRunningTime="2026-07-22 18:48:13.227758981 +0000 UTC m=+2047.443406385"
Jul 22 18:48:17.474823 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:17.474786    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:48:19.219693 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:19.219664    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:23.430559 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.430524    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"]
Jul 22 18:48:23.434261 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.434240    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:23.436455 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.436436    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:48:23.441584 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.441565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"]
Jul 22 18:48:23.586117 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.586083    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x\" (UID: \"d68b73c6-fd42-4af2-a4b4-c7501edc213b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:23.686663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.686580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x\" (UID: \"d68b73c6-fd42-4af2-a4b4-c7501edc213b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:23.687205 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.687185    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x\" (UID: \"d68b73c6-fd42-4af2-a4b4-c7501edc213b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:23.747791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:23.747766    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:24.079683 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:24.079659    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"]
Jul 22 18:48:24.081847 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:48:24.081819    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68b73c6_fd42_4af2_a4b4_c7501edc213b.slice/crio-f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3 WatchSource:0}: Error finding container f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3: Status 404 returned error can't find the container with id f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3
Jul 22 18:48:24.267448 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:24.267421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x" event={"ID":"d68b73c6-fd42-4af2-a4b4-c7501edc213b","Type":"ContainerStarted","Data":"f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3"}
Jul 22 18:48:25.274210 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:25.274175    2580 generic.go:356] "Generic (PLEG): container finished" podID="d68b73c6-fd42-4af2-a4b4-c7501edc213b" containerID="68bd3587b4167defc52fada4ba2eacdbcfaa84310c5bbe4e0a16a63ef65bd4bc" exitCode=0
Jul 22 18:48:25.274659 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:25.274263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x" event={"ID":"d68b73c6-fd42-4af2-a4b4-c7501edc213b","Type":"ContainerDied","Data":"68bd3587b4167defc52fada4ba2eacdbcfaa84310c5bbe4e0a16a63ef65bd4bc"}
Jul 22 18:48:26.408183 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:26.408159    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:26.509866 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:26.509841    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca\") pod \"d68b73c6-fd42-4af2-a4b4-c7501edc213b\" (UID: \"d68b73c6-fd42-4af2-a4b4-c7501edc213b\") "
Jul 22 18:48:26.510163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:26.510142    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d68b73c6-fd42-4af2-a4b4-c7501edc213b" (UID: "d68b73c6-fd42-4af2-a4b4-c7501edc213b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:48:26.611295 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:26.611236    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68b73c6-fd42-4af2-a4b4-c7501edc213b-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:48:27.285232 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:27.285200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x" event={"ID":"d68b73c6-fd42-4af2-a4b4-c7501edc213b","Type":"ContainerDied","Data":"f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3"}
Jul 22 18:48:27.285397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:27.285235    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"
Jul 22 18:48:27.285397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:27.285274    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f88ab6064793bb22587132692f4996d22d9901d405a5e931f984a24cbc660ff3"
Jul 22 18:48:34.118316 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.118127    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:34.118738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.118373    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" containerName="maas-api" containerID="cri-o://8ecbf668a9f8db40469e36ec7110a3ff2f9ddf9aafb525484051f2aad53454d1" gracePeriod=30
Jul 22 18:48:34.214816 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.214777    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.55:8443/health\": dial tcp 10.134.0.55:8443: connect: connection refused"
Jul 22 18:48:34.323621 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.323592    2580 generic.go:356] "Generic (PLEG): container finished" podID="c2acbf4d-7570-41ab-8343-ad244777f543" containerID="8ecbf668a9f8db40469e36ec7110a3ff2f9ddf9aafb525484051f2aad53454d1" exitCode=0
Jul 22 18:48:34.323778 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.323657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" event={"ID":"c2acbf4d-7570-41ab-8343-ad244777f543","Type":"ContainerDied","Data":"8ecbf668a9f8db40469e36ec7110a3ff2f9ddf9aafb525484051f2aad53454d1"}
Jul 22 18:48:34.362329 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.362308    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:34.373898 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.373839    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca\") pod \"c2acbf4d-7570-41ab-8343-ad244777f543\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") "
Jul 22 18:48:34.374009 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.373924    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls\") pod \"c2acbf4d-7570-41ab-8343-ad244777f543\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") "
Jul 22 18:48:34.374009 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.373959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp\") pod \"c2acbf4d-7570-41ab-8343-ad244777f543\" (UID: \"c2acbf4d-7570-41ab-8343-ad244777f543\") "
Jul 22 18:48:34.374187 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.374162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2acbf4d-7570-41ab-8343-ad244777f543" (UID: "c2acbf4d-7570-41ab-8343-ad244777f543"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:48:34.375817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.375774    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp" (OuterVolumeSpecName: "kube-api-access-kdxqp") pod "c2acbf4d-7570-41ab-8343-ad244777f543" (UID: "c2acbf4d-7570-41ab-8343-ad244777f543"). InnerVolumeSpecName "kube-api-access-kdxqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:48:34.375817 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.375788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2acbf4d-7570-41ab-8343-ad244777f543" (UID: "c2acbf4d-7570-41ab-8343-ad244777f543"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:48:34.474668 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.474641    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2acbf4d-7570-41ab-8343-ad244777f543-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:48:34.474826 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.474675    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdxqp\" (UniqueName: \"kubernetes.io/projected/c2acbf4d-7570-41ab-8343-ad244777f543-kube-api-access-kdxqp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:48:34.474826 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:34.474689    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2acbf4d-7570-41ab-8343-ad244777f543-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:48:35.329371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:35.329340    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz" event={"ID":"c2acbf4d-7570-41ab-8343-ad244777f543","Type":"ContainerDied","Data":"c2de36ba76ca0f63ec850f91344a0b026d8c393592533d7883c9907d022b1cd7"}
Jul 22 18:48:35.329371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:35.329362    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"
Jul 22 18:48:35.329852 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:35.329393    2580 scope.go:117] "RemoveContainer" containerID="8ecbf668a9f8db40469e36ec7110a3ff2f9ddf9aafb525484051f2aad53454d1"
Jul 22 18:48:35.351436 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:35.351410    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:35.355285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:35.355266    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d85e8a51-db4b7d858-rzzcz"]
Jul 22 18:48:36.319977 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:36.319948    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" path="/var/lib/kubelet/pods/c2acbf4d-7570-41ab-8343-ad244777f543/volumes"
Jul 22 18:48:37.075525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:37.075495    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:48:38.513452 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:38.513416    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"]
Jul 22 18:48:38.515819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:38.515783    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d85e8a51-n5n5x"]
Jul 22 18:48:40.319731 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:48:40.319696    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d68b73c6-fd42-4af2-a4b4-c7501edc213b" path="/var/lib/kubelet/pods/d68b73c6-fd42-4af2-a4b4-c7501edc213b/volumes"
Jul 22 18:49:08.107479 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.107440    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:08.107999 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.107982    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" containerName="maas-api"
Jul 22 18:49:08.108079 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.108002    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" containerName="maas-api"
Jul 22 18:49:08.108079 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.108019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d68b73c6-fd42-4af2-a4b4-c7501edc213b" containerName="revoke-keys"
Jul 22 18:49:08.108079 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.108028    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d68b73c6-fd42-4af2-a4b4-c7501edc213b" containerName="revoke-keys"
Jul 22 18:49:08.108230 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.108144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d68b73c6-fd42-4af2-a4b4-c7501edc213b" containerName="revoke-keys"
Jul 22 18:49:08.108230 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.108160    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2acbf4d-7570-41ab-8343-ad244777f543" containerName="maas-api"
Jul 22 18:49:08.112009 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.111989    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.116517 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.116498    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f02c5841-serving-cert\""
Jul 22 18:49:08.116626 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.116540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:49:08.120195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.120172    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:08.257552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.257515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.257552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.257555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h6db\" (UniqueName: \"kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.257783 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.257576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.359218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.359127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.359218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.359171    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h6db\" (UniqueName: \"kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.359218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.359194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.359505 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:49:08.359368    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-serving-cert: secret "maas-api-e2e-mt-f02c5841-serving-cert" not found
Jul 22 18:49:08.359505 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:49:08.359434    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls podName:77fdc488-d341-4b27-942f-9c15c15049c9 nodeName:}" failed. No retries permitted until 2026-07-22 18:49:08.859412531 +0000 UTC m=+2103.075059912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls") pod "maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" (UID: "77fdc488-d341-4b27-942f-9c15c15049c9") : secret "maas-api-e2e-mt-f02c5841-serving-cert" not found
Jul 22 18:49:08.359800 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.359779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.367749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.367729    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h6db\" (UniqueName: \"kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.864906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.864872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:08.867245 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:08.867219    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") pod \"maas-api-e2e-mt-f02c5841-6557cffd98-j56ll\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:09.023317 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:09.023287    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:09.155367 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:09.155340    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:09.157903 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:49:09.157874    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77fdc488_d341_4b27_942f_9c15c15049c9.slice/crio-dbe196155c0441f351530456bd08b2d4d9248943e15700ed8abe12fe6eb16e1f WatchSource:0}: Error finding container dbe196155c0441f351530456bd08b2d4d9248943e15700ed8abe12fe6eb16e1f: Status 404 returned error can't find the container with id dbe196155c0441f351530456bd08b2d4d9248943e15700ed8abe12fe6eb16e1f
Jul 22 18:49:09.159232 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:09.159211    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:49:09.499736 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:09.499697    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" event={"ID":"77fdc488-d341-4b27-942f-9c15c15049c9","Type":"ContainerStarted","Data":"dbe196155c0441f351530456bd08b2d4d9248943e15700ed8abe12fe6eb16e1f"}
Jul 22 18:49:10.506218 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:10.506176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" event={"ID":"77fdc488-d341-4b27-942f-9c15c15049c9","Type":"ContainerStarted","Data":"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"}
Jul 22 18:49:10.506632 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:10.506320    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:10.523296 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:10.523248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" podStartSLOduration=2.1770773820000002 podStartE2EDuration="2.523232792s" totalImagesPullingTime="346.15541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:10.521865014 +0000 UTC m=+2104.737512416" watchObservedRunningTime="2026-07-22 18:49:10.523232792 +0000 UTC m=+2104.738880195"
Jul 22 18:49:12.473543 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:12.473508    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:49:16.515954 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:16.515914    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:22.492552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.492516    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:49:22.495764 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.495748    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.503512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.502944    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f02c5841\"/\"openshift-service-ca.crt\""
Jul 22 18:49:22.503512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.503066    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-f02c5841\"/\"kube-root-ca.crt\""
Jul 22 18:49:22.503512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.503093    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f02c5841\"/\"default-dockercfg-hpb7t\""
Jul 22 18:49:22.503512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.503375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-f02c5841\"/\"e2e-lifecycle-model-f02c5841-kserve-self-signed-certs\""
Jul 22 18:49:22.512194 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.512173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:49:22.577371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577345    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.577524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577386    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfgp\" (UniqueName: \"kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.577524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.577524 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.577686 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.577686 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.577617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.678927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.678897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679078 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.678935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679078 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.678978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679078 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679017    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfgp\" (UniqueName: \"kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679078 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679071    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679268 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679357 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679335    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679398 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.679572 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.679545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.681370 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.681346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.681471 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.681440    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.686761 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.686737    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfgp\" (UniqueName: \"kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp\") pod \"e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") " pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.806583 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.806521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:22.935373 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:22.935347    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:49:22.937450 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:49:22.937410    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9ea2ca8_d3a6_4807_b962_ca833e8575cc.slice/crio-776545d6fb09d45f432d3087ecb0590aeac933477522f5890849dbb28e2b5674 WatchSource:0}: Error finding container 776545d6fb09d45f432d3087ecb0590aeac933477522f5890849dbb28e2b5674: Status 404 returned error can't find the container with id 776545d6fb09d45f432d3087ecb0590aeac933477522f5890849dbb28e2b5674
Jul 22 18:49:23.569572 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:23.569535    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" event={"ID":"b9ea2ca8-d3a6-4807-b962-ca833e8575cc","Type":"ContainerStarted","Data":"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"}
Jul 22 18:49:23.569985 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:23.569581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" event={"ID":"b9ea2ca8-d3a6-4807-b962-ca833e8575cc","Type":"ContainerStarted","Data":"776545d6fb09d45f432d3087ecb0590aeac933477522f5890849dbb28e2b5674"}
Jul 22 18:49:23.585646 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:23.585598    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" podStartSLOduration=1.5855833160000001 podStartE2EDuration="1.585583316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:23.58468079 +0000 UTC m=+2117.800328214" watchObservedRunningTime="2026-07-22 18:49:23.585583316 +0000 UTC m=+2117.801231045"
Jul 22 18:49:25.763337 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:25.763304    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:49:32.807376 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:32.807331    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:32.807900 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:32.807392    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:32.819698 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:32.819675    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:33.629738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:33.629704    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:49:36.483556 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:36.483518    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:49:40.801743 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.801706    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"]
Jul 22 18:49:40.805197 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.805179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:40.807369 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.807349    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:49:40.811397 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.811376    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"]
Jul 22 18:49:40.819674 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.819643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f02c5841-c2htg\" (UID: \"f86fbf43-772d-46d9-9317-ebcd8ad51be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:40.920625 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.920592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f02c5841-c2htg\" (UID: \"f86fbf43-772d-46d9-9317-ebcd8ad51be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:40.921287 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:40.921262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f02c5841-c2htg\" (UID: \"f86fbf43-772d-46d9-9317-ebcd8ad51be6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:41.118259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:41.118168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:41.247409 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:41.247386    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"]
Jul 22 18:49:41.249947 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:49:41.249926    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86fbf43_772d_46d9_9317_ebcd8ad51be6.slice/crio-2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45 WatchSource:0}: Error finding container 2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45: Status 404 returned error can't find the container with id 2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45
Jul 22 18:49:41.658512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:41.658478    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg" event={"ID":"f86fbf43-772d-46d9-9317-ebcd8ad51be6","Type":"ContainerStarted","Data":"f62b8676eb7447d7e6288e01bd46abd98c3f4d1551367835e597518f5cd373c0"}
Jul 22 18:49:41.658673 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:41.658534    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg" event={"ID":"f86fbf43-772d-46d9-9317-ebcd8ad51be6","Type":"ContainerStarted","Data":"2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45"}
Jul 22 18:49:41.673337 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:41.673293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg" podStartSLOduration=1.673277579 podStartE2EDuration="1.673277579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:49:41.671891837 +0000 UTC m=+2135.887539239" watchObservedRunningTime="2026-07-22 18:49:41.673277579 +0000 UTC m=+2135.888924981"
Jul 22 18:49:42.665251 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:42.665215    2580 generic.go:356] "Generic (PLEG): container finished" podID="f86fbf43-772d-46d9-9317-ebcd8ad51be6" containerID="f62b8676eb7447d7e6288e01bd46abd98c3f4d1551367835e597518f5cd373c0" exitCode=0
Jul 22 18:49:42.665621 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:42.665290    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg" event={"ID":"f86fbf43-772d-46d9-9317-ebcd8ad51be6","Type":"ContainerDied","Data":"f62b8676eb7447d7e6288e01bd46abd98c3f4d1551367835e597518f5cd373c0"}
Jul 22 18:49:43.802384 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:43.802359    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:43.843269 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:43.843245    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca\") pod \"f86fbf43-772d-46d9-9317-ebcd8ad51be6\" (UID: \"f86fbf43-772d-46d9-9317-ebcd8ad51be6\") "
Jul 22 18:49:43.843573 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:43.843552    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f86fbf43-772d-46d9-9317-ebcd8ad51be6" (UID: "f86fbf43-772d-46d9-9317-ebcd8ad51be6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:49:43.944124 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:43.944096    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f86fbf43-772d-46d9-9317-ebcd8ad51be6-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:49:44.676588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:44.676562    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"
Jul 22 18:49:44.676768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:44.676593    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg" event={"ID":"f86fbf43-772d-46d9-9317-ebcd8ad51be6","Type":"ContainerDied","Data":"2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45"}
Jul 22 18:49:44.676768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:44.676650    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2acb9e991979ebd837980b7af2d90f30913df3623032e60a4a59d9f584c5fa45"
Jul 22 18:49:54.876703 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:54.876665    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:49:56.241738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.241702    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:56.242219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.242045    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" podUID="77fdc488-d341-4b27-942f-9c15c15049c9" containerName="maas-api" containerID="cri-o://551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c" gracePeriod=30
Jul 22 18:49:56.486116 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.486096    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:56.548510 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.548437    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca\") pod \"77fdc488-d341-4b27-942f-9c15c15049c9\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") "
Jul 22 18:49:56.548510 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.548477    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") pod \"77fdc488-d341-4b27-942f-9c15c15049c9\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") "
Jul 22 18:49:56.548697 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.548511    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h6db\" (UniqueName: \"kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db\") pod \"77fdc488-d341-4b27-942f-9c15c15049c9\" (UID: \"77fdc488-d341-4b27-942f-9c15c15049c9\") "
Jul 22 18:49:56.548842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.548799    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "77fdc488-d341-4b27-942f-9c15c15049c9" (UID: "77fdc488-d341-4b27-942f-9c15c15049c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:49:56.550596 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.550574    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db" (OuterVolumeSpecName: "kube-api-access-6h6db") pod "77fdc488-d341-4b27-942f-9c15c15049c9" (UID: "77fdc488-d341-4b27-942f-9c15c15049c9"). InnerVolumeSpecName "kube-api-access-6h6db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:49:56.550707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.550687    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "77fdc488-d341-4b27-942f-9c15c15049c9" (UID: "77fdc488-d341-4b27-942f-9c15c15049c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:49:56.649672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.649633    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77fdc488-d341-4b27-942f-9c15c15049c9-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:49:56.649672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.649664    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77fdc488-d341-4b27-942f-9c15c15049c9-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:49:56.649672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.649674    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h6db\" (UniqueName: \"kubernetes.io/projected/77fdc488-d341-4b27-942f-9c15c15049c9-kube-api-access-6h6db\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:49:56.737359 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.737322    2580 generic.go:356] "Generic (PLEG): container finished" podID="77fdc488-d341-4b27-942f-9c15c15049c9" containerID="551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c" exitCode=0
Jul 22 18:49:56.737525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.737375    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"
Jul 22 18:49:56.737525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.737407    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" event={"ID":"77fdc488-d341-4b27-942f-9c15c15049c9","Type":"ContainerDied","Data":"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"}
Jul 22 18:49:56.737525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.737463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll" event={"ID":"77fdc488-d341-4b27-942f-9c15c15049c9","Type":"ContainerDied","Data":"dbe196155c0441f351530456bd08b2d4d9248943e15700ed8abe12fe6eb16e1f"}
Jul 22 18:49:56.737525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.737487    2580 scope.go:117] "RemoveContainer" containerID="551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"
Jul 22 18:49:56.749690 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.749664    2580 scope.go:117] "RemoveContainer" containerID="551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"
Jul 22 18:49:56.750388 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:49:56.750357    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c\": container with ID starting with 551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c not found: ID does not exist" containerID="551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"
Jul 22 18:49:56.750496 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.750398    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c"} err="failed to get container status \"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c\": rpc error: code = NotFound desc = could not find container \"551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c\": container with ID starting with 551f323867c2ccc4b29da1a1b5042edb8c440b44659942dc1015cc2a8a521e5c not found: ID does not exist"
Jul 22 18:49:56.763394 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.763368    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:56.767366 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:56.767347    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f02c5841-6557cffd98-j56ll"]
Jul 22 18:49:58.319910 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:49:58.319828    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77fdc488-d341-4b27-942f-9c15c15049c9" path="/var/lib/kubelet/pods/77fdc488-d341-4b27-942f-9c15c15049c9/volumes"
Jul 22 18:50:00.903146 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:00.903113    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"]
Jul 22 18:50:00.906981 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:00.906957    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f02c5841-c2htg"]
Jul 22 18:50:02.320687 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:02.320654    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f86fbf43-772d-46d9-9317-ebcd8ad51be6" path="/var/lib/kubelet/pods/f86fbf43-772d-46d9-9317-ebcd8ad51be6/volumes"
Jul 22 18:50:05.472282 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:05.472252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:50:06.419959 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:06.419928    2580 scope.go:117] "RemoveContainer" containerID="dc49ecb2fee1ed40a530333707e532d737bedacd5f2d9b13385e7b486d2b1fed"
Jul 22 18:50:08.226398 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:08.226366    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:50:23.246911 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.246847    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" podUID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" containerName="main" containerID="cri-o://cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd" gracePeriod=45
Jul 22 18:50:23.494015 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.493992    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:50:23.571615 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571538    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.571615 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571611    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.571819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571629    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.571819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwfgp\" (UniqueName: \"kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.571819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.571819 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571730    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache\") pod \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\" (UID: \"b9ea2ca8-d3a6-4807-b962-ca833e8575cc\") "
Jul 22 18:50:23.572045 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.571910    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:50:23.572045 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.572017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home" (OuterVolumeSpecName: "home") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:50:23.572145 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.572044    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:50:23.573677 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.573654    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:50:23.573773 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.573695    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm" (OuterVolumeSpecName: "dshm") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:50:23.573773 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.573731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp" (OuterVolumeSpecName: "kube-api-access-kwfgp") pod "b9ea2ca8-d3a6-4807-b962-ca833e8575cc" (UID: "b9ea2ca8-d3a6-4807-b962-ca833e8575cc"). InnerVolumeSpecName "kube-api-access-kwfgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:50:23.673135 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673097    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.673135 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673133    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.673135 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673143    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfgp\" (UniqueName: \"kubernetes.io/projected/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-kube-api-access-kwfgp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.673343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673152    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.673343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673162    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.673343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.673169    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9ea2ca8-d3a6-4807-b962-ca833e8575cc-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:50:23.868046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.867975    2580 generic.go:356] "Generic (PLEG): container finished" podID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" containerID="cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd" exitCode=0
Jul 22 18:50:23.868046 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.868014    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"
Jul 22 18:50:23.868212 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.868056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" event={"ID":"b9ea2ca8-d3a6-4807-b962-ca833e8575cc","Type":"ContainerDied","Data":"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"}
Jul 22 18:50:23.868212 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.868096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr" event={"ID":"b9ea2ca8-d3a6-4807-b962-ca833e8575cc","Type":"ContainerDied","Data":"776545d6fb09d45f432d3087ecb0590aeac933477522f5890849dbb28e2b5674"}
Jul 22 18:50:23.868212 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.868112    2580 scope.go:117] "RemoveContainer" containerID="cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"
Jul 22 18:50:23.877060 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.877043    2580 scope.go:117] "RemoveContainer" containerID="cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"
Jul 22 18:50:23.877289 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:50:23.877272    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd\": container with ID starting with cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd not found: ID does not exist" containerID="cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"
Jul 22 18:50:23.877360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.877296    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd"} err="failed to get container status \"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd\": rpc error: code = NotFound desc = could not find container \"cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd\": container with ID starting with cf3e773126fddf67f94c69cea5474e2767c7d887b4298e7153d91ac5d0dfe7fd not found: ID does not exist"
Jul 22 18:50:23.892623 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.892599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:50:23.897303 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:23.897285    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-f02c5841/e2e-lifecycle-model-f02c5841-kserve-6c7d6bbf47-k65jr"]
Jul 22 18:50:24.320197 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:50:24.320157    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" path="/var/lib/kubelet/pods/b9ea2ca8-d3a6-4807-b962-ca833e8575cc/volumes"
Jul 22 18:51:01.784444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:01.784400    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:51:06.443042 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:06.443010    2580 scope.go:117] "RemoveContainer" containerID="f52773d5eea728df85abea37716d217ae0835ef69a21a2a62a00376928e50d1a"
Jul 22 18:51:14.178691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:14.178660    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:51:55.874942 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.874864    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875231    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77fdc488-d341-4b27-942f-9c15c15049c9" containerName="maas-api"
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875242    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="77fdc488-d341-4b27-942f-9c15c15049c9" containerName="maas-api"
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875259    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f86fbf43-772d-46d9-9317-ebcd8ad51be6" containerName="revoke-keys"
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875265    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f86fbf43-772d-46d9-9317-ebcd8ad51be6" containerName="revoke-keys"
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875280    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" containerName="main"
Jul 22 18:51:55.875326 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875286    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" containerName="main"
Jul 22 18:51:55.875511 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875350    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f86fbf43-772d-46d9-9317-ebcd8ad51be6" containerName="revoke-keys"
Jul 22 18:51:55.875511 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875360    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="77fdc488-d341-4b27-942f-9c15c15049c9" containerName="maas-api"
Jul 22 18:51:55.875511 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.875367    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ea2ca8-d3a6-4807-b962-ca833e8575cc" containerName="main"
Jul 22 18:51:55.878195 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.878179    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:55.881122 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.881087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d001ad-serving-cert\""
Jul 22 18:51:55.881122 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.881105    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:51:55.890240 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:55.890210    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:51:56.020181 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.020146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8m8\" (UniqueName: \"kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.020352 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.020198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.020352 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.020280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.121356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.121317    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.121535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.121372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8m8\" (UniqueName: \"kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.121535 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.121415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.121623 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:51:56.121541    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-serving-cert: secret "maas-api-e2e-inf-a-d001ad-serving-cert" not found
Jul 22 18:51:56.121623 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:51:56.121608    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls podName:0c74b326-5f5f-471e-984a-d7f7b2cab6af nodeName:}" failed. No retries permitted until 2026-07-22 18:51:56.621587189 +0000 UTC m=+2270.837234587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls") pod "maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" (UID: "0c74b326-5f5f-471e-984a-d7f7b2cab6af") : secret "maas-api-e2e-inf-a-d001ad-serving-cert" not found
Jul 22 18:51:56.121980 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.121960    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.135957 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.135898    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8m8\" (UniqueName: \"kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.625434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.625394    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.627870 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.627852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") pod \"maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.789532 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.789494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:56.921786 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:56.921761    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:51:56.923514 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:51:56.923476    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c74b326_5f5f_471e_984a_d7f7b2cab6af.slice/crio-cdb8c11e1fb3826b522bdc038d8c0be90474b3bbeb8284c1b86c65803195b3c0 WatchSource:0}: Error finding container cdb8c11e1fb3826b522bdc038d8c0be90474b3bbeb8284c1b86c65803195b3c0: Status 404 returned error can't find the container with id cdb8c11e1fb3826b522bdc038d8c0be90474b3bbeb8284c1b86c65803195b3c0
Jul 22 18:51:57.328433 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:57.328398    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" event={"ID":"0c74b326-5f5f-471e-984a-d7f7b2cab6af","Type":"ContainerStarted","Data":"cdb8c11e1fb3826b522bdc038d8c0be90474b3bbeb8284c1b86c65803195b3c0"}
Jul 22 18:51:58.335418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:58.335372    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" event={"ID":"0c74b326-5f5f-471e-984a-d7f7b2cab6af","Type":"ContainerStarted","Data":"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"}
Jul 22 18:51:58.335841 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:58.335532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:51:58.359454 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:51:58.356900    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" podStartSLOduration=2.859161325 podStartE2EDuration="3.356879006s" totalImagesPullingTime="497.717681ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:51:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:51:58.352538602 +0000 UTC m=+2272.568186005" watchObservedRunningTime="2026-07-22 18:51:58.356879006 +0000 UTC m=+2272.572526411"
Jul 22 18:52:01.272557 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:01.272523    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:52:03.921788 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.921743    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:52:03.925627 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.925602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:03.929236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.929202    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ef44fb-serving-cert\""
Jul 22 18:52:03.938896 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.938873    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:52:03.989498 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.989467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:03.989684 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.989517    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:03.989684 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:03.989552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lr2h\" (UniqueName: \"kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.090829 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.090769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.091019 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.090841    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.091019 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.090882    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lr2h\" (UniqueName: \"kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.091019 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:52:04.090990    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-serving-cert: secret "maas-api-e2e-inf-b-ef44fb-serving-cert" not found
Jul 22 18:52:04.091162 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:52:04.091100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls podName:98bf083a-55dd-4ad5-8e83-ab129dfdee22 nodeName:}" failed. No retries permitted until 2026-07-22 18:52:04.591076619 +0000 UTC m=+2278.806724017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls") pod "maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" (UID: "98bf083a-55dd-4ad5-8e83-ab129dfdee22") : secret "maas-api-e2e-inf-b-ef44fb-serving-cert" not found
Jul 22 18:52:04.091518 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.091501    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.105116 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.105085    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lr2h\" (UniqueName: \"kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.345605 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.345577    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:52:04.594966 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.594930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.597511 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.597447    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") pod \"maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.838652 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.838617    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:04.969612 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:04.969584    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:52:04.972365 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:52:04.972332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bf083a_55dd_4ad5_8e83_ab129dfdee22.slice/crio-61df5dd5d490fc7aa09fa6a392aa6d053c9f5ba235e60d56eecc7d4cee1c6407 WatchSource:0}: Error finding container 61df5dd5d490fc7aa09fa6a392aa6d053c9f5ba235e60d56eecc7d4cee1c6407: Status 404 returned error can't find the container with id 61df5dd5d490fc7aa09fa6a392aa6d053c9f5ba235e60d56eecc7d4cee1c6407
Jul 22 18:52:05.368316 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:05.368280    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" event={"ID":"98bf083a-55dd-4ad5-8e83-ab129dfdee22","Type":"ContainerStarted","Data":"61df5dd5d490fc7aa09fa6a392aa6d053c9f5ba235e60d56eecc7d4cee1c6407"}
Jul 22 18:52:06.374488 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:06.374447    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" event={"ID":"98bf083a-55dd-4ad5-8e83-ab129dfdee22","Type":"ContainerStarted","Data":"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"}
Jul 22 18:52:06.374908 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:06.374564    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:06.391366 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:06.391319    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" podStartSLOduration=2.496872233 podStartE2EDuration="3.391303453s" totalImagesPullingTime="894.43122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:52:06.389217848 +0000 UTC m=+2280.604865273" watchObservedRunningTime="2026-07-22 18:52:06.391303453 +0000 UTC m=+2280.606950856"
Jul 22 18:52:06.466353 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:06.466319    2580 scope.go:117] "RemoveContainer" containerID="f1a59e27e091a7bc57a14dd1dd4a64697afa8358ef4fa476caf9edaf09c20599"
Jul 22 18:52:09.984728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:09.984694    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:52:12.385290 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.385261    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:52:12.783048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.783009    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:52:12.786622 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.786601    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.789077 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.789055    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d001ad\"/\"kube-root-ca.crt\""
Jul 22 18:52:12.789791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.789774    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d001ad\"/\"openshift-service-ca.crt\""
Jul 22 18:52:12.789878 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.789785    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d001ad\"/\"test-model-d001ad-kserve-self-signed-certs\""
Jul 22 18:52:12.789878 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.789776    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d001ad\"/\"default-dockercfg-mcnxb\""
Jul 22 18:52:12.797780 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.797757    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:52:12.864638 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.864842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864668    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.864842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.864842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864716    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.864842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhrz\" (UniqueName: \"kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.864842 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.864836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965673 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965869 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965869 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965708    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965869 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965726    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965869 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhrz\" (UniqueName: \"kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.965869 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.965775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.966132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.966108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.966132 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.966125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.966205 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.966173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.968183 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.968151    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.968372 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.968358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:12.972719 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:12.972694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhrz\" (UniqueName: \"kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz\") pod \"test-model-d001ad-kserve-5899ffcdc7-kgtsh\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") " pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:13.096193 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:13.096106    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:13.231893 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:13.231849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:52:13.234550 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:52:13.234511    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb00cadeb_dd8d_4551_9ca9_df9d3cc31761.slice/crio-6c42c87e6626f9e41e323df38cf57ebbc441b8c576997f25ab21a851517910ea WatchSource:0}: Error finding container 6c42c87e6626f9e41e323df38cf57ebbc441b8c576997f25ab21a851517910ea: Status 404 returned error can't find the container with id 6c42c87e6626f9e41e323df38cf57ebbc441b8c576997f25ab21a851517910ea
Jul 22 18:52:13.409788 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:13.409692    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" event={"ID":"b00cadeb-dd8d-4551-9ca9-df9d3cc31761","Type":"ContainerStarted","Data":"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"}
Jul 22 18:52:13.409788 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:13.409739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" event={"ID":"b00cadeb-dd8d-4551-9ca9-df9d3cc31761","Type":"ContainerStarted","Data":"6c42c87e6626f9e41e323df38cf57ebbc441b8c576997f25ab21a851517910ea"}
Jul 22 18:52:13.427333 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:13.427284    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" podStartSLOduration=1.427269102 podStartE2EDuration="1.427269102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:52:13.426264479 +0000 UTC m=+2287.641911904" watchObservedRunningTime="2026-07-22 18:52:13.427269102 +0000 UTC m=+2287.642916551"
Jul 22 18:52:23.097227 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:23.097189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:23.097227 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:23.097235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:23.109650 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:23.109619    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:23.270672 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:23.270640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:52:23.472688 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:23.472660    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:52:40.804927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.804891    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:52:40.809741 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.809717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.813207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.813185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef44fb\"/\"default-dockercfg-jxf5v\""
Jul 22 18:52:40.813207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.813185    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef44fb\"/\"test-model-ef44fb-kserve-self-signed-certs\""
Jul 22 18:52:40.813404 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.813187    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef44fb\"/\"openshift-service-ca.crt\""
Jul 22 18:52:40.813507 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.813493    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef44fb\"/\"kube-root-ca.crt\""
Jul 22 18:52:40.816903 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.816880    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:52:40.907068 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907032    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.907260 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.907260 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907153    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.907260 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.907390 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:40.907390 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:40.907290    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z655p\" (UniqueName: \"kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008547    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008565    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z655p\" (UniqueName: \"kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008593    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.008712 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.009032 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.008973    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.009162 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.009072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.009162 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.009079    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.010979 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.010956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.011206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.011189    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.016731 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.016708    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z655p\" (UniqueName: \"kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p\") pod \"test-model-ef44fb-kserve-5984f77b59-klhnw\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") " pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.121084 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.120997    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:41.253629 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.253602    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:52:41.256138 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:52:41.256112    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d67f3ba_b325_4250_b1ec_18d08676ecd3.slice/crio-2a54a31fcc3a9c18d2b30474522658c21cd1efd0bc298498966bd6b9da8c4bd0 WatchSource:0}: Error finding container 2a54a31fcc3a9c18d2b30474522658c21cd1efd0bc298498966bd6b9da8c4bd0: Status 404 returned error can't find the container with id 2a54a31fcc3a9c18d2b30474522658c21cd1efd0bc298498966bd6b9da8c4bd0
Jul 22 18:52:41.549536 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.549497    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" event={"ID":"6d67f3ba-b325-4250-b1ec-18d08676ecd3","Type":"ContainerStarted","Data":"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"}
Jul 22 18:52:41.549702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.549545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" event={"ID":"6d67f3ba-b325-4250-b1ec-18d08676ecd3","Type":"ContainerStarted","Data":"2a54a31fcc3a9c18d2b30474522658c21cd1efd0bc298498966bd6b9da8c4bd0"}
Jul 22 18:52:41.567844 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:41.567779    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" podStartSLOduration=1.5677646269999999 podStartE2EDuration="1.567764627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:52:41.566398285 +0000 UTC m=+2315.782045686" watchObservedRunningTime="2026-07-22 18:52:41.567764627 +0000 UTC m=+2315.783412030"
Jul 22 18:52:44.774343 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:44.774303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:52:51.121718 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:51.121678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:51.122238 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:51.121741    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:51.134123 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:51.134094    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:52:51.613055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:52:51.613022    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:53:01.367950 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:01.367860    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:53:06.486622 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:06.486584    2580 scope.go:117] "RemoveContainer" containerID="afd1227457880e180280352c27c1168caedadf07d1bb81c8279d4bd3736bc5db"
Jul 22 18:53:22.940902 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:22.940871    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:53:23.846348 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:23.846314    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:53:27.965101 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:27.965065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:53:34.552861 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:34.552828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:53:34.553297 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:34.553047    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" podUID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" containerName="maas-api" containerID="cri-o://3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b" gracePeriod=30
Jul 22 18:53:35.002859 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.002835    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:53:35.162525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.162460    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") pod \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") "
Jul 22 18:53:35.162525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.162500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs8m8\" (UniqueName: \"kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8\") pod \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") "
Jul 22 18:53:35.162693 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.162577    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca\") pod \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\" (UID: \"0c74b326-5f5f-471e-984a-d7f7b2cab6af\") "
Jul 22 18:53:35.162944 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.162914    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c74b326-5f5f-471e-984a-d7f7b2cab6af" (UID: "0c74b326-5f5f-471e-984a-d7f7b2cab6af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:53:35.164437 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.164417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8" (OuterVolumeSpecName: "kube-api-access-cs8m8") pod "0c74b326-5f5f-471e-984a-d7f7b2cab6af" (UID: "0c74b326-5f5f-471e-984a-d7f7b2cab6af"). InnerVolumeSpecName "kube-api-access-cs8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:53:35.164518 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.164464    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c74b326-5f5f-471e-984a-d7f7b2cab6af" (UID: "0c74b326-5f5f-471e-984a-d7f7b2cab6af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:53:35.263644 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.263617    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c74b326-5f5f-471e-984a-d7f7b2cab6af-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:35.263644 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.263640    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c74b326-5f5f-471e-984a-d7f7b2cab6af-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:35.263794 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.263653    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs8m8\" (UniqueName: \"kubernetes.io/projected/0c74b326-5f5f-471e-984a-d7f7b2cab6af-kube-api-access-cs8m8\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:35.822253 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.822217    2580 generic.go:356] "Generic (PLEG): container finished" podID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" containerID="3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b" exitCode=0
Jul 22 18:53:35.822671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.822267    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"
Jul 22 18:53:35.822671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.822301    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" event={"ID":"0c74b326-5f5f-471e-984a-d7f7b2cab6af","Type":"ContainerDied","Data":"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"}
Jul 22 18:53:35.822671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.822354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z" event={"ID":"0c74b326-5f5f-471e-984a-d7f7b2cab6af","Type":"ContainerDied","Data":"cdb8c11e1fb3826b522bdc038d8c0be90474b3bbeb8284c1b86c65803195b3c0"}
Jul 22 18:53:35.822671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.822375    2580 scope.go:117] "RemoveContainer" containerID="3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"
Jul 22 18:53:35.833249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.833231    2580 scope.go:117] "RemoveContainer" containerID="3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"
Jul 22 18:53:35.833534 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:53:35.833516    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b\": container with ID starting with 3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b not found: ID does not exist" containerID="3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"
Jul 22 18:53:35.833588 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.833542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b"} err="failed to get container status \"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b\": rpc error: code = NotFound desc = could not find container \"3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b\": container with ID starting with 3291b048b6d5a313df16b72574e7a046b78f0722f5874f1163b1c24f500f4f1b not found: ID does not exist"
Jul 22 18:53:35.844366 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.844343    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:53:35.847697 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:35.847677    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d001ad-67d4448fdf-2cx6z"]
Jul 22 18:53:36.320320 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:36.320289    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" path="/var/lib/kubelet/pods/0c74b326-5f5f-471e-984a-d7f7b2cab6af/volumes"
Jul 22 18:53:37.960852 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:37.960792    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" podUID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" containerName="main" containerID="cri-o://efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84" gracePeriod=45
Jul 22 18:53:38.208120 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.208098    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:53:38.293190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293122    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293186    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293229    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckhrz\" (UniqueName: \"kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293301    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293325    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home" (OuterVolumeSpecName: "home") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:38.293407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293374    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm\") pod \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\" (UID: \"b00cadeb-dd8d-4551-9ca9-df9d3cc31761\") "
Jul 22 18:53:38.293630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293556    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:38.293630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293572    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache" (OuterVolumeSpecName: "model-cache") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:38.293697 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293662    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.293697 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293682    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.293771 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.293696    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.295464 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.295431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz" (OuterVolumeSpecName: "kube-api-access-ckhrz") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "kube-api-access-ckhrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:53:38.295591 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.295460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:53:38.295591 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.295488    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm" (OuterVolumeSpecName: "dshm") pod "b00cadeb-dd8d-4551-9ca9-df9d3cc31761" (UID: "b00cadeb-dd8d-4551-9ca9-df9d3cc31761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:38.394306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.394277    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.394306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.394300    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhrz\" (UniqueName: \"kubernetes.io/projected/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-kube-api-access-ckhrz\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.394306 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.394308    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b00cadeb-dd8d-4551-9ca9-df9d3cc31761-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:38.838714 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.838676    2580 generic.go:356] "Generic (PLEG): container finished" podID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" containerID="efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84" exitCode=0
Jul 22 18:53:38.838912 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.838723    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"
Jul 22 18:53:38.838912 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.838723    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" event={"ID":"b00cadeb-dd8d-4551-9ca9-df9d3cc31761","Type":"ContainerDied","Data":"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"}
Jul 22 18:53:38.838912 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.838769    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh" event={"ID":"b00cadeb-dd8d-4551-9ca9-df9d3cc31761","Type":"ContainerDied","Data":"6c42c87e6626f9e41e323df38cf57ebbc441b8c576997f25ab21a851517910ea"}
Jul 22 18:53:38.838912 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.838788    2580 scope.go:117] "RemoveContainer" containerID="efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"
Jul 22 18:53:38.847209 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.847182    2580 scope.go:117] "RemoveContainer" containerID="efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"
Jul 22 18:53:38.847458 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:53:38.847435    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84\": container with ID starting with efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84 not found: ID does not exist" containerID="efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"
Jul 22 18:53:38.847521 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.847466    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84"} err="failed to get container status \"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84\": rpc error: code = NotFound desc = could not find container \"efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84\": container with ID starting with efb2cd908780f7d6f0edcee7ba85839f343cf5e51365076415067195f5d37d84 not found: ID does not exist"
Jul 22 18:53:38.855707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.855685    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:53:38.859427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.859408    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d001ad/test-model-d001ad-kserve-5899ffcdc7-kgtsh"]
Jul 22 18:53:38.865878 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:38.865854    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" podUID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" containerName="main" containerID="cri-o://0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d" gracePeriod=45
Jul 22 18:53:39.113520 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.113496    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:53:39.200246 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z655p\" (UniqueName: \"kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200262    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200297    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200314    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200360 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200349    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200567 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache\") pod \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\" (UID: \"6d67f3ba-b325-4250-b1ec-18d08676ecd3\") "
Jul 22 18:53:39.200640 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200578    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:39.200707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home" (OuterVolumeSpecName: "home") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:39.200757 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.200719    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:39.202371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.202342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:53:39.202371 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.202340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p" (OuterVolumeSpecName: "kube-api-access-z655p") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "kube-api-access-z655p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:53:39.202494 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.202434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm" (OuterVolumeSpecName: "dshm") pod "6d67f3ba-b325-4250-b1ec-18d08676ecd3" (UID: "6d67f3ba-b325-4250-b1ec-18d08676ecd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:53:39.301657 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301622    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z655p\" (UniqueName: \"kubernetes.io/projected/6d67f3ba-b325-4250-b1ec-18d08676ecd3-kube-api-access-z655p\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.301657 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301656    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.301657 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301666    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.301879 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301675    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.301879 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301682    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.301879 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.301690    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d67f3ba-b325-4250-b1ec-18d08676ecd3-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:39.845591 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.845553    2580 generic.go:356] "Generic (PLEG): container finished" podID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" containerID="0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d" exitCode=0
Jul 22 18:53:39.845795 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.845604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"
Jul 22 18:53:39.845795 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.845644    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" event={"ID":"6d67f3ba-b325-4250-b1ec-18d08676ecd3","Type":"ContainerDied","Data":"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"}
Jul 22 18:53:39.845795 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.845689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw" event={"ID":"6d67f3ba-b325-4250-b1ec-18d08676ecd3","Type":"ContainerDied","Data":"2a54a31fcc3a9c18d2b30474522658c21cd1efd0bc298498966bd6b9da8c4bd0"}
Jul 22 18:53:39.845795 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.845713    2580 scope.go:117] "RemoveContainer" containerID="0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"
Jul 22 18:53:39.854757 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.854739    2580 scope.go:117] "RemoveContainer" containerID="0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"
Jul 22 18:53:39.855072 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:53:39.855055    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d\": container with ID starting with 0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d not found: ID does not exist" containerID="0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"
Jul 22 18:53:39.855136 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.855082    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d"} err="failed to get container status \"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d\": rpc error: code = NotFound desc = could not find container \"0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d\": container with ID starting with 0b4193771ab4de7005df34a62d930200514baaccce0e7980707ca142d7eb529d not found: ID does not exist"
Jul 22 18:53:39.865527 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.865504    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:53:39.874406 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:39.874385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef44fb/test-model-ef44fb-kserve-5984f77b59-klhnw"]
Jul 22 18:53:40.321174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:40.321136    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" path="/var/lib/kubelet/pods/6d67f3ba-b325-4250-b1ec-18d08676ecd3/volumes"
Jul 22 18:53:40.321548 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:40.321513    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" path="/var/lib/kubelet/pods/b00cadeb-dd8d-4551-9ca9-df9d3cc31761/volumes"
Jul 22 18:53:43.672227 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:43.672187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:53:46.121317 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121284    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"]
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121623    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" containerName="maas-api"
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121633    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" containerName="maas-api"
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121653    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" containerName="main"
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121658    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" containerName="main"
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121666    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" containerName="main"
Jul 22 18:53:46.121691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121671    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" containerName="main"
Jul 22 18:53:46.121897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121730    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d67f3ba-b325-4250-b1ec-18d08676ecd3" containerName="main"
Jul 22 18:53:46.121897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121743    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b00cadeb-dd8d-4551-9ca9-df9d3cc31761" containerName="main"
Jul 22 18:53:46.121897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.121749    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c74b326-5f5f-471e-984a-d7f7b2cab6af" containerName="maas-api"
Jul 22 18:53:46.124662 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.124646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:46.127055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.127032    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:53:46.131569 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.131543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"]
Jul 22 18:53:46.266515 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.266479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb\" (UID: \"caec9429-d796-474d-9d63-9f32148c824c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:46.367504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.367471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb\" (UID: \"caec9429-d796-474d-9d63-9f32148c824c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:46.368103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.368083    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb\" (UID: \"caec9429-d796-474d-9d63-9f32148c824c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:46.435613 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.435578    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:46.567344 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.567314    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"]
Jul 22 18:53:46.569332 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:53:46.569310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaec9429_d796_474d_9d63_9f32148c824c.slice/crio-d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f WatchSource:0}: Error finding container d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f: Status 404 returned error can't find the container with id d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f
Jul 22 18:53:46.882418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.882378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb" event={"ID":"caec9429-d796-474d-9d63-9f32148c824c","Type":"ContainerStarted","Data":"cc71bd0a8ba982b39e22a01e97f90d5d503f5089006fd9d8dcdc67c64482ac8a"}
Jul 22 18:53:46.882663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.882635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb" event={"ID":"caec9429-d796-474d-9d63-9f32148c824c","Type":"ContainerStarted","Data":"d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f"}
Jul 22 18:53:46.899795 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:46.899751    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb" podStartSLOduration=0.899736206 podStartE2EDuration="899.736206ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:53:46.898646901 +0000 UTC m=+2381.114294304" watchObservedRunningTime="2026-07-22 18:53:46.899736206 +0000 UTC m=+2381.115383662"
Jul 22 18:53:47.888933 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:47.888901    2580 generic.go:356] "Generic (PLEG): container finished" podID="caec9429-d796-474d-9d63-9f32148c824c" containerID="cc71bd0a8ba982b39e22a01e97f90d5d503f5089006fd9d8dcdc67c64482ac8a" exitCode=0
Jul 22 18:53:47.889325 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:47.888990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb" event={"ID":"caec9429-d796-474d-9d63-9f32148c824c","Type":"ContainerDied","Data":"cc71bd0a8ba982b39e22a01e97f90d5d503f5089006fd9d8dcdc67c64482ac8a"}
Jul 22 18:53:49.027970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.027940    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:49.190129 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.190104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca\") pod \"caec9429-d796-474d-9d63-9f32148c824c\" (UID: \"caec9429-d796-474d-9d63-9f32148c824c\") "
Jul 22 18:53:49.190471 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.190447    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caec9429-d796-474d-9d63-9f32148c824c" (UID: "caec9429-d796-474d-9d63-9f32148c824c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:53:49.291091 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.291055    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caec9429-d796-474d-9d63-9f32148c824c-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:49.899774 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.899739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb" event={"ID":"caec9429-d796-474d-9d63-9f32148c824c","Type":"ContainerDied","Data":"d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f"}
Jul 22 18:53:49.899774 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.899765    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"
Jul 22 18:53:49.899988 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:49.899789    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c80f1c4b5a6eec146dc6c909744b3b97aa4a21b97fececa44ccd6c8c90218f"
Jul 22 18:53:56.645858 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.645791    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:53:56.646659 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.646064    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" podUID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" containerName="maas-api" containerID="cri-o://21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c" gracePeriod=30
Jul 22 18:53:56.897605 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.897543    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:53:56.937241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.937211    2580 generic.go:356] "Generic (PLEG): container finished" podID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" containerID="21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c" exitCode=0
Jul 22 18:53:56.937427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.937270    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"
Jul 22 18:53:56.937427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.937294    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" event={"ID":"98bf083a-55dd-4ad5-8e83-ab129dfdee22","Type":"ContainerDied","Data":"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"}
Jul 22 18:53:56.937427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.937356    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h" event={"ID":"98bf083a-55dd-4ad5-8e83-ab129dfdee22","Type":"ContainerDied","Data":"61df5dd5d490fc7aa09fa6a392aa6d053c9f5ba235e60d56eecc7d4cee1c6407"}
Jul 22 18:53:56.937427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.937387    2580 scope.go:117] "RemoveContainer" containerID="21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"
Jul 22 18:53:56.945742 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.945720    2580 scope.go:117] "RemoveContainer" containerID="21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"
Jul 22 18:53:56.946015 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:53:56.945995    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c\": container with ID starting with 21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c not found: ID does not exist" containerID="21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"
Jul 22 18:53:56.946071 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.946024    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c"} err="failed to get container status \"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c\": rpc error: code = NotFound desc = could not find container \"21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c\": container with ID starting with 21bd76a720b69c776f7e463f5d838f5b5742c9687d82e5d75be226f5c66d786c not found: ID does not exist"
Jul 22 18:53:56.953425 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.953410    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca\") pod \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") "
Jul 22 18:53:56.953486 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.953455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") pod \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") "
Jul 22 18:53:56.953486 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.953473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lr2h\" (UniqueName: \"kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h\") pod \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\" (UID: \"98bf083a-55dd-4ad5-8e83-ab129dfdee22\") "
Jul 22 18:53:56.953749 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.953731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98bf083a-55dd-4ad5-8e83-ab129dfdee22" (UID: "98bf083a-55dd-4ad5-8e83-ab129dfdee22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:53:56.955453 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.955431    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98bf083a-55dd-4ad5-8e83-ab129dfdee22" (UID: "98bf083a-55dd-4ad5-8e83-ab129dfdee22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:53:56.955533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:56.955478    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h" (OuterVolumeSpecName: "kube-api-access-2lr2h") pod "98bf083a-55dd-4ad5-8e83-ab129dfdee22" (UID: "98bf083a-55dd-4ad5-8e83-ab129dfdee22"). InnerVolumeSpecName "kube-api-access-2lr2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:53:57.054689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:57.054654    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98bf083a-55dd-4ad5-8e83-ab129dfdee22-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:57.054689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:57.054683    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98bf083a-55dd-4ad5-8e83-ab129dfdee22-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:57.054689 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:57.054692    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lr2h\" (UniqueName: \"kubernetes.io/projected/98bf083a-55dd-4ad5-8e83-ab129dfdee22-kube-api-access-2lr2h\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:53:57.258221 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:57.258189    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:53:57.266082 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:57.266053    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ef44fb-78bf76bfcf-lvv7h"]
Jul 22 18:53:58.321980 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:58.321947    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" path="/var/lib/kubelet/pods/98bf083a-55dd-4ad5-8e83-ab129dfdee22/volumes"
Jul 22 18:53:59.473546 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:53:59.473512    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:54:01.205207 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:01.205172    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"]
Jul 22 18:54:01.208136 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:01.208111    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ef44fb-tjnnb"]
Jul 22 18:54:02.320256 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:02.320219    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caec9429-d796-474d-9d63-9f32148c824c" path="/var/lib/kubelet/pods/caec9429-d796-474d-9d63-9f32148c824c/volumes"
Jul 22 18:54:11.542130 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542097    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:11.542523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542470    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caec9429-d796-474d-9d63-9f32148c824c" containerName="revoke-keys"
Jul 22 18:54:11.542523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542481    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="caec9429-d796-474d-9d63-9f32148c824c" containerName="revoke-keys"
Jul 22 18:54:11.542523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542490    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" containerName="maas-api"
Jul 22 18:54:11.542523 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542497    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" containerName="maas-api"
Jul 22 18:54:11.542674 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542545    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="caec9429-d796-474d-9d63-9f32148c824c" containerName="revoke-keys"
Jul 22 18:54:11.542674 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.542555    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="98bf083a-55dd-4ad5-8e83-ab129dfdee22" containerName="maas-api"
Jul 22 18:54:11.547133 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.547113    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.551011 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.550990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f44f9a-serving-cert\""
Jul 22 18:54:11.551081 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.551000    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:54:11.556975 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.556952    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:11.575145 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.575115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.575294 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.575151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwlp\" (UniqueName: \"kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.575294 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.575185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.676193 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.676158    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.676193 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.676194    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwlp\" (UniqueName: \"kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.676389 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.676224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.676389 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:54:11.676351    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-serving-cert: secret "maas-api-e2e-api-a-f44f9a-serving-cert" not found
Jul 22 18:54:11.676465 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:54:11.676421    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls podName:ba03e757-8b39-4974-ab10-b91f22189628 nodeName:}" failed. No retries permitted until 2026-07-22 18:54:12.176404089 +0000 UTC m=+2406.392051488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls") pod "maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" (UID: "ba03e757-8b39-4974-ab10-b91f22189628") : secret "maas-api-e2e-api-a-f44f9a-serving-cert" not found
Jul 22 18:54:11.676872 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.676854    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:11.685054 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:11.685023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwlp\" (UniqueName: \"kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:12.180039 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:12.180002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:12.182518 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:12.182499    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") pod \"maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:12.457995 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:12.457918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:12.615303 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:12.615276    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:12.616985 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:54:12.616950    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba03e757_8b39_4974_ab10_b91f22189628.slice/crio-36e70c587b54384b6950c127d8e1201a16de1f72f21d7ec10047aeb0c6c2ca4c WatchSource:0}: Error finding container 36e70c587b54384b6950c127d8e1201a16de1f72f21d7ec10047aeb0c6c2ca4c: Status 404 returned error can't find the container with id 36e70c587b54384b6950c127d8e1201a16de1f72f21d7ec10047aeb0c6c2ca4c
Jul 22 18:54:12.618185 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:12.618167    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 18:54:13.016463 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:13.016430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" event={"ID":"ba03e757-8b39-4974-ab10-b91f22189628","Type":"ContainerStarted","Data":"36e70c587b54384b6950c127d8e1201a16de1f72f21d7ec10047aeb0c6c2ca4c"}
Jul 22 18:54:13.967887 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:13.967848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:54:14.023520 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:14.023485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" event={"ID":"ba03e757-8b39-4974-ab10-b91f22189628","Type":"ContainerStarted","Data":"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"}
Jul 22 18:54:14.023692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:14.023551    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:14.041249 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:14.041201    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" podStartSLOduration=2.647637956 podStartE2EDuration="3.041184669s" totalImagesPullingTime="393.546713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:54:14.039121375 +0000 UTC m=+2408.254768781" watchObservedRunningTime="2026-07-22 18:54:14.041184669 +0000 UTC m=+2408.256832072"
Jul 22 18:54:19.574866 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.571915    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:54:19.578925 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.578895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.581569 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.581543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-9984f4-serving-cert\""
Jul 22 18:54:19.583789 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.583763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:54:19.640850 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.640824    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc4tl\" (UniqueName: \"kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.640970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.640856    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.640970 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.640887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.742138 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.742111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc4tl\" (UniqueName: \"kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.742236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.742142    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.742236 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.742176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.742311 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:54:19.742276    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-serving-cert: secret "maas-api-e2e-api-b-9984f4-serving-cert" not found
Jul 22 18:54:19.742346 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:54:19.742338    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls podName:f0646207-8d32-4779-a13e-74ce280aa67b nodeName:}" failed. No retries permitted until 2026-07-22 18:54:20.242322682 +0000 UTC m=+2414.457970063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls") pod "maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" (UID: "f0646207-8d32-4779-a13e-74ce280aa67b") : secret "maas-api-e2e-api-b-9984f4-serving-cert" not found
Jul 22 18:54:19.742789 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.742772    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:19.752686 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:19.752660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc4tl\" (UniqueName: \"kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:20.034884 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:20.034856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:20.245454 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:20.245426    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:20.247630 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:20.247608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") pod \"maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:20.490590 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:20.490558    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:20.618787 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:20.618759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:54:20.620363 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:54:20.620338    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0646207_8d32_4779_a13e_74ce280aa67b.slice/crio-b5da1b9c145bc7bce3943d7e74ed76192356fed7909408606e73cb85b822d73e WatchSource:0}: Error finding container b5da1b9c145bc7bce3943d7e74ed76192356fed7909408606e73cb85b822d73e: Status 404 returned error can't find the container with id b5da1b9c145bc7bce3943d7e74ed76192356fed7909408606e73cb85b822d73e
Jul 22 18:54:21.058114 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:21.058041    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" event={"ID":"f0646207-8d32-4779-a13e-74ce280aa67b","Type":"ContainerStarted","Data":"b5da1b9c145bc7bce3943d7e74ed76192356fed7909408606e73cb85b822d73e"}
Jul 22 18:54:22.064626 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:22.064588    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" event={"ID":"f0646207-8d32-4779-a13e-74ce280aa67b","Type":"ContainerStarted","Data":"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"}
Jul 22 18:54:22.065104 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:22.064654    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:22.084445 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:22.084400    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" podStartSLOduration=2.6618051510000003 podStartE2EDuration="3.084385296s" totalImagesPullingTime="422.580145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:54:22.082353479 +0000 UTC m=+2416.298000902" watchObservedRunningTime="2026-07-22 18:54:22.084385296 +0000 UTC m=+2416.300032698"
Jul 22 18:54:25.094533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:25.094500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:54:28.074389 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:28.074316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:54:32.072089 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.072050    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"]
Jul 22 18:54:32.075903 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.075884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:32.078145 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.078119    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:54:32.092568 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.092543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"]
Jul 22 18:54:32.147669 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.147637    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw\" (UID: \"cba76894-4eaf-4788-bd83-f691847f5c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:32.248490 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.248463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw\" (UID: \"cba76894-4eaf-4788-bd83-f691847f5c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:32.249070 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.249052    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw\" (UID: \"cba76894-4eaf-4788-bd83-f691847f5c98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:32.394717 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.394655    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:32.527168 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:32.527146    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"]
Jul 22 18:54:32.529416 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:54:32.529388    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba76894_4eaf_4788_bd83_f691847f5c98.slice/crio-11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e WatchSource:0}: Error finding container 11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e: Status 404 returned error can't find the container with id 11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e
Jul 22 18:54:33.123701 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:33.123609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw" event={"ID":"cba76894-4eaf-4788-bd83-f691847f5c98","Type":"ContainerStarted","Data":"7401efe84dd40322f8f2e2cfaf5237f9edc5d66f67c5fe4482e6e9758784e105"}
Jul 22 18:54:33.123701 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:33.123668    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw" event={"ID":"cba76894-4eaf-4788-bd83-f691847f5c98","Type":"ContainerStarted","Data":"11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e"}
Jul 22 18:54:33.137543 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:33.137486    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw" podStartSLOduration=1.137467132 podStartE2EDuration="1.137467132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:54:33.136933666 +0000 UTC m=+2427.352581069" watchObservedRunningTime="2026-07-22 18:54:33.137467132 +0000 UTC m=+2427.353114540"
Jul 22 18:54:34.129605 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:34.129572    2580 generic.go:356] "Generic (PLEG): container finished" podID="cba76894-4eaf-4788-bd83-f691847f5c98" containerID="7401efe84dd40322f8f2e2cfaf5237f9edc5d66f67c5fe4482e6e9758784e105" exitCode=0
Jul 22 18:54:34.130023 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:34.129667    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw" event={"ID":"cba76894-4eaf-4788-bd83-f691847f5c98","Type":"ContainerDied","Data":"7401efe84dd40322f8f2e2cfaf5237f9edc5d66f67c5fe4482e6e9758784e105"}
Jul 22 18:54:35.263848 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:35.263798    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:35.375582 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:35.375555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca\") pod \"cba76894-4eaf-4788-bd83-f691847f5c98\" (UID: \"cba76894-4eaf-4788-bd83-f691847f5c98\") "
Jul 22 18:54:35.375914 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:35.375893    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cba76894-4eaf-4788-bd83-f691847f5c98" (UID: "cba76894-4eaf-4788-bd83-f691847f5c98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:54:35.476335 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:35.476313    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cba76894-4eaf-4788-bd83-f691847f5c98-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:54:36.145599 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:36.145446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw" event={"ID":"cba76894-4eaf-4788-bd83-f691847f5c98","Type":"ContainerDied","Data":"11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e"}
Jul 22 18:54:36.145775 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:36.145604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"
Jul 22 18:54:36.145775 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:36.145649    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11f49f64043fb54ea1a3cb20490df83a2ea256d1baca4c80eb9466dbba15df4e"
Jul 22 18:54:47.875225 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:47.875192    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:47.875699 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:47.875428    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" podUID="ba03e757-8b39-4974-ab10-b91f22189628" containerName="maas-api" containerID="cri-o://8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723" gracePeriod=30
Jul 22 18:54:48.122037 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.122012    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:48.206511 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.206479    2580 generic.go:356] "Generic (PLEG): container finished" podID="ba03e757-8b39-4974-ab10-b91f22189628" containerID="8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723" exitCode=0
Jul 22 18:54:48.206724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.206534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"
Jul 22 18:54:48.206724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.206573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" event={"ID":"ba03e757-8b39-4974-ab10-b91f22189628","Type":"ContainerDied","Data":"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"}
Jul 22 18:54:48.206724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.206632    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg" event={"ID":"ba03e757-8b39-4974-ab10-b91f22189628","Type":"ContainerDied","Data":"36e70c587b54384b6950c127d8e1201a16de1f72f21d7ec10047aeb0c6c2ca4c"}
Jul 22 18:54:48.206724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.206658    2580 scope.go:117] "RemoveContainer" containerID="8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"
Jul 22 18:54:48.214937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.214921    2580 scope.go:117] "RemoveContainer" containerID="8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"
Jul 22 18:54:48.215185 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:54:48.215166    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723\": container with ID starting with 8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723 not found: ID does not exist" containerID="8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"
Jul 22 18:54:48.215242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.215199    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723"} err="failed to get container status \"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723\": rpc error: code = NotFound desc = could not find container \"8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723\": container with ID starting with 8e48811070acc47c19d1ced05c64827bbf05e246a971425702d73e9321e81723 not found: ID does not exist"
Jul 22 18:54:48.286028 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.285996    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca\") pod \"ba03e757-8b39-4974-ab10-b91f22189628\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") "
Jul 22 18:54:48.286206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.286120    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") pod \"ba03e757-8b39-4974-ab10-b91f22189628\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") "
Jul 22 18:54:48.286206 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.286141    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kwlp\" (UniqueName: \"kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp\") pod \"ba03e757-8b39-4974-ab10-b91f22189628\" (UID: \"ba03e757-8b39-4974-ab10-b91f22189628\") "
Jul 22 18:54:48.286362 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.286339    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ba03e757-8b39-4974-ab10-b91f22189628" (UID: "ba03e757-8b39-4974-ab10-b91f22189628"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:54:48.288305 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.288274    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp" (OuterVolumeSpecName: "kube-api-access-7kwlp") pod "ba03e757-8b39-4974-ab10-b91f22189628" (UID: "ba03e757-8b39-4974-ab10-b91f22189628"). InnerVolumeSpecName "kube-api-access-7kwlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:54:48.288429 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.288317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ba03e757-8b39-4974-ab10-b91f22189628" (UID: "ba03e757-8b39-4974-ab10-b91f22189628"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:54:48.387618 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.387580    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba03e757-8b39-4974-ab10-b91f22189628-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:54:48.387618 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.387617    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwlp\" (UniqueName: \"kubernetes.io/projected/ba03e757-8b39-4974-ab10-b91f22189628-kube-api-access-7kwlp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:54:48.387843 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.387632    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba03e757-8b39-4974-ab10-b91f22189628-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:54:48.522757 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.522680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:48.526972 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:48.526949    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f44f9a-694c64f5d8-pc5qg"]
Jul 22 18:54:50.273275 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:50.273240    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:54:50.320816 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:50.320770    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba03e757-8b39-4974-ab10-b91f22189628" path="/var/lib/kubelet/pods/ba03e757-8b39-4974-ab10-b91f22189628/volumes"
Jul 22 18:54:52.155715 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:52.155680    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"]
Jul 22 18:54:52.157493 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:52.157471    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f44f9a-m2nhw"]
Jul 22 18:54:52.320309 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:52.320275    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba76894-4eaf-4788-bd83-f691847f5c98" path="/var/lib/kubelet/pods/cba76894-4eaf-4788-bd83-f691847f5c98/volumes"
Jul 22 18:54:58.974253 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974216    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"]
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974584    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba03e757-8b39-4974-ab10-b91f22189628" containerName="maas-api"
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974596    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba03e757-8b39-4974-ab10-b91f22189628" containerName="maas-api"
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974606    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba76894-4eaf-4788-bd83-f691847f5c98" containerName="revoke-keys"
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974612    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba76894-4eaf-4788-bd83-f691847f5c98" containerName="revoke-keys"
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974669    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba03e757-8b39-4974-ab10-b91f22189628" containerName="maas-api"
Jul 22 18:54:58.974728 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.974680    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba76894-4eaf-4788-bd83-f691847f5c98" containerName="revoke-keys"
Jul 22 18:54:58.978792 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.978775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:54:58.981003 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.980977    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 18:54:58.984778 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:58.984752    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"]
Jul 22 18:54:59.078428 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:59.078395    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9984f4-t2n97\" (UID: \"7074a08b-1d66-41cb-a7ea-491f4407c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:54:59.179533 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:59.179496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9984f4-t2n97\" (UID: \"7074a08b-1d66-41cb-a7ea-491f4407c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:54:59.180174 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:59.180152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9984f4-t2n97\" (UID: \"7074a08b-1d66-41cb-a7ea-491f4407c503\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:54:59.290529 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:59.290427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:54:59.434891 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:54:59.434859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"]
Jul 22 18:54:59.436965 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:54:59.436933    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7074a08b_1d66_41cb_a7ea_491f4407c503.slice/crio-7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8 WatchSource:0}: Error finding container 7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8: Status 404 returned error can't find the container with id 7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8
Jul 22 18:55:00.268852 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:00.268789    2580 generic.go:356] "Generic (PLEG): container finished" podID="7074a08b-1d66-41cb-a7ea-491f4407c503" containerID="89a2e7bfbce527c4eec299350e1e5617f7cb4944e957c8a33a6b4588573e03c0" exitCode=0
Jul 22 18:55:00.269257 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:00.268876    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97" event={"ID":"7074a08b-1d66-41cb-a7ea-491f4407c503","Type":"ContainerDied","Data":"89a2e7bfbce527c4eec299350e1e5617f7cb4944e957c8a33a6b4588573e03c0"}
Jul 22 18:55:00.269257 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:00.268938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97" event={"ID":"7074a08b-1d66-41cb-a7ea-491f4407c503","Type":"ContainerStarted","Data":"7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8"}
Jul 22 18:55:01.406701 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:01.406674    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:55:01.500481 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:01.500441    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca\") pod \"7074a08b-1d66-41cb-a7ea-491f4407c503\" (UID: \"7074a08b-1d66-41cb-a7ea-491f4407c503\") "
Jul 22 18:55:01.500801 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:01.500775    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7074a08b-1d66-41cb-a7ea-491f4407c503" (UID: "7074a08b-1d66-41cb-a7ea-491f4407c503"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:55:01.601916 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:01.601832    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7074a08b-1d66-41cb-a7ea-491f4407c503-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:55:02.279542 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:02.279507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97" event={"ID":"7074a08b-1d66-41cb-a7ea-491f4407c503","Type":"ContainerDied","Data":"7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8"}
Jul 22 18:55:02.279744 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:02.279562    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7447bbb22d5b8c3305f4dce9ba1109f88ff606d3ac53814e9239d11c70c226b8"
Jul 22 18:55:02.279744 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:02.279523    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"
Jul 22 18:55:06.526908 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:06.526875    2580 scope.go:117] "RemoveContainer" containerID="68bd3587b4167defc52fada4ba2eacdbcfaa84310c5bbe4e0a16a63ef65bd4bc"
Jul 22 18:55:14.404407 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.404376    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:55:14.404883 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.404575    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" podUID="f0646207-8d32-4779-a13e-74ce280aa67b" containerName="maas-api" containerID="cri-o://8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9" gracePeriod=30
Jul 22 18:55:14.653668 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.653638    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:55:14.718057 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.718026    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca\") pod \"f0646207-8d32-4779-a13e-74ce280aa67b\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") "
Jul 22 18:55:14.718224 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.718067    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc4tl\" (UniqueName: \"kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl\") pod \"f0646207-8d32-4779-a13e-74ce280aa67b\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") "
Jul 22 18:55:14.718224 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.718132    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") pod \"f0646207-8d32-4779-a13e-74ce280aa67b\" (UID: \"f0646207-8d32-4779-a13e-74ce280aa67b\") "
Jul 22 18:55:14.718439 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.718420    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0646207-8d32-4779-a13e-74ce280aa67b" (UID: "f0646207-8d32-4779-a13e-74ce280aa67b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 18:55:14.720200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.720171    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0646207-8d32-4779-a13e-74ce280aa67b" (UID: "f0646207-8d32-4779-a13e-74ce280aa67b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:55:14.720312 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.720264    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl" (OuterVolumeSpecName: "kube-api-access-pc4tl") pod "f0646207-8d32-4779-a13e-74ce280aa67b" (UID: "f0646207-8d32-4779-a13e-74ce280aa67b"). InnerVolumeSpecName "kube-api-access-pc4tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:55:14.819495 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.819465    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc4tl\" (UniqueName: \"kubernetes.io/projected/f0646207-8d32-4779-a13e-74ce280aa67b-kube-api-access-pc4tl\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:55:14.819495 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.819491    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0646207-8d32-4779-a13e-74ce280aa67b-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:55:14.819495 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:14.819500    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0646207-8d32-4779-a13e-74ce280aa67b-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:55:15.343055 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.343023    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0646207-8d32-4779-a13e-74ce280aa67b" containerID="8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9" exitCode=0
Jul 22 18:55:15.343241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.343068    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"
Jul 22 18:55:15.343241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.343112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" event={"ID":"f0646207-8d32-4779-a13e-74ce280aa67b","Type":"ContainerDied","Data":"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"}
Jul 22 18:55:15.343241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.343177    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m" event={"ID":"f0646207-8d32-4779-a13e-74ce280aa67b","Type":"ContainerDied","Data":"b5da1b9c145bc7bce3943d7e74ed76192356fed7909408606e73cb85b822d73e"}
Jul 22 18:55:15.343241 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.343207    2580 scope.go:117] "RemoveContainer" containerID="8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"
Jul 22 18:55:15.351763 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.351744    2580 scope.go:117] "RemoveContainer" containerID="8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"
Jul 22 18:55:15.352028 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:55:15.352009    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9\": container with ID starting with 8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9 not found: ID does not exist" containerID="8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"
Jul 22 18:55:15.352103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.352036    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9"} err="failed to get container status \"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9\": rpc error: code = NotFound desc = could not find container \"8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9\": container with ID starting with 8692ce98d62271a72eb4be0238eae89b11d405ca75082613d8f367a761f75bd9 not found: ID does not exist"
Jul 22 18:55:15.364821 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.364782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:55:15.368203 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:15.368182    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-9984f4-75c58d9bf7-clc5m"]
Jul 22 18:55:16.321219 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:16.321188    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0646207-8d32-4779-a13e-74ce280aa67b" path="/var/lib/kubelet/pods/f0646207-8d32-4779-a13e-74ce280aa67b/volumes"
Jul 22 18:55:18.969285 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:18.969254    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:55:19.083702 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:19.083669    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"]
Jul 22 18:55:19.097224 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:19.097193    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9984f4-t2n97"]
Jul 22 18:55:20.321865 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:20.321830    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7074a08b-1d66-41cb-a7ea-491f4407c503" path="/var/lib/kubelet/pods/7074a08b-1d66-41cb-a7ea-491f4407c503/volumes"
Jul 22 18:55:29.944661 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.944630    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 18:55:29.945075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945004    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0646207-8d32-4779-a13e-74ce280aa67b" containerName="maas-api"
Jul 22 18:55:29.945075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945018    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0646207-8d32-4779-a13e-74ce280aa67b" containerName="maas-api"
Jul 22 18:55:29.945075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945026    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7074a08b-1d66-41cb-a7ea-491f4407c503" containerName="revoke-keys"
Jul 22 18:55:29.945075 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945032    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7074a08b-1d66-41cb-a7ea-491f4407c503" containerName="revoke-keys"
Jul 22 18:55:29.945220 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945082    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0646207-8d32-4779-a13e-74ce280aa67b" containerName="maas-api"
Jul 22 18:55:29.945220 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.945088    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7074a08b-1d66-41cb-a7ea-491f4407c503" containerName="revoke-keys"
Jul 22 18:55:29.949170 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.949152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:29.951867 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.951783    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-d24283-serving-cert\""
Jul 22 18:55:29.951996 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.951870    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wl92n\""
Jul 22 18:55:29.959242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:29.959212    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 18:55:30.054175 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.054143    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq7lp\" (UniqueName: \"kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.054345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.054198    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.054345 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.054255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.155525 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.155495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.155768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.155548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.155768 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.155591    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq7lp\" (UniqueName: \"kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.156228 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.156202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.157924 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.157905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.167293 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.167270    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq7lp\" (UniqueName: \"kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp\") pod \"maas-api-e2e-shared-a-d24283-788454d79d-287hw\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.259752 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.259652    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:30.393040 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.393013    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 18:55:30.395587 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:55:30.395553    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8cbadd_c040_446d_bf72_12b7a3c5cfe7.slice/crio-0303e1bea438fd92c49a3f463d667ce5c56c22ea2f7a79e440a25591a6f102db WatchSource:0}: Error finding container 0303e1bea438fd92c49a3f463d667ce5c56c22ea2f7a79e440a25591a6f102db: Status 404 returned error can't find the container with id 0303e1bea438fd92c49a3f463d667ce5c56c22ea2f7a79e440a25591a6f102db
Jul 22 18:55:30.413980 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:30.413951    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" event={"ID":"df8cbadd-c040-446d-bf72-12b7a3c5cfe7","Type":"ContainerStarted","Data":"0303e1bea438fd92c49a3f463d667ce5c56c22ea2f7a79e440a25591a6f102db"}
Jul 22 18:55:31.420332 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:31.420243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" event={"ID":"df8cbadd-c040-446d-bf72-12b7a3c5cfe7","Type":"ContainerStarted","Data":"125e154c215e811055a7f5fd241885f0636d3f11c3488eba6e0d42bdd2ec842d"}
Jul 22 18:55:31.420332 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:31.420328    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:31.437552 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:31.437502    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" podStartSLOduration=1.82219892 podStartE2EDuration="2.43748797s" totalImagesPullingTime="615.28905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:55:31.436029223 +0000 UTC m=+2485.651676649" watchObservedRunningTime="2026-07-22 18:55:31.43748797 +0000 UTC m=+2485.653135415"
Jul 22 18:55:36.873898 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:36.873865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:55:37.430276 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:37.430245    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 18:55:37.987675 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:37.987643    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 18:55:37.992053 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:37.992031    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:37.994478 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:37.994455    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-63c82d-serving-cert\""
Jul 22 18:55:38.001746 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.001720    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 18:55:38.120862 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.120828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvfv\" (UniqueName: \"kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.121027 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.120888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.121027 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.120999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.221504 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.221471    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znvfv\" (UniqueName: \"kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.221671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.221533    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.221671 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.221615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.221750 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:55:38.221689    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-serving-cert: secret "maas-api-e2e-shared-b-63c82d-serving-cert" not found
Jul 22 18:55:38.221782 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:55:38.221749    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls podName:111e1044-6683-459f-9e1f-3e0cb9b8fd4c nodeName:}" failed. No retries permitted until 2026-07-22 18:55:38.72173279 +0000 UTC m=+2492.937380175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls") pod "maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" (UID: "111e1044-6683-459f-9e1f-3e0cb9b8fd4c") : secret "maas-api-e2e-shared-b-63c82d-serving-cert" not found
Jul 22 18:55:38.222211 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.222194    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.233265 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.233232    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvfv\" (UniqueName: \"kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.725382 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.725341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.727821 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.727781    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") pod \"maas-api-e2e-shared-b-63c82d-886559bb-d8z6q\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:38.903259 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:38.903216    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:39.036625 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:39.036603    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 18:55:39.039754 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:55:39.039723    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111e1044_6683_459f_9e1f_3e0cb9b8fd4c.slice/crio-6aef4afe3b0d610d1d801a22789e278b0db316db740c2bcddf140b9b8f42546b WatchSource:0}: Error finding container 6aef4afe3b0d610d1d801a22789e278b0db316db740c2bcddf140b9b8f42546b: Status 404 returned error can't find the container with id 6aef4afe3b0d610d1d801a22789e278b0db316db740c2bcddf140b9b8f42546b
Jul 22 18:55:39.458727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:39.458684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" event={"ID":"111e1044-6683-459f-9e1f-3e0cb9b8fd4c","Type":"ContainerStarted","Data":"6aef4afe3b0d610d1d801a22789e278b0db316db740c2bcddf140b9b8f42546b"}
Jul 22 18:55:40.464692 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:40.464654    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" event={"ID":"111e1044-6683-459f-9e1f-3e0cb9b8fd4c","Type":"ContainerStarted","Data":"ba5a241944de83f357c13852a22e0d75c302739841266ae6cd4d177771312dd6"}
Jul 22 18:55:40.465198 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:40.464781    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:40.482138 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:40.482092    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" podStartSLOduration=2.70730874 podStartE2EDuration="3.482075505s" totalImagesPullingTime="774.766765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:55:40.480717178 +0000 UTC m=+2494.696364583" watchObservedRunningTime="2026-07-22 18:55:40.482075505 +0000 UTC m=+2494.697722908"
Jul 22 18:55:46.475315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:46.475287    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 18:55:47.968617 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:55:47.968586    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:56:02.495997 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.495906    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:56:02.500146 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.500125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.503644 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.503621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"default-dockercfg-fsm6p\""
Jul 22 18:56:02.503785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.503641    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"kube-root-ca.crt\""
Jul 22 18:56:02.503785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.503672    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"openshift-service-ca.crt\""
Jul 22 18:56:02.503785 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.503621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"auth-test-model-d24283-kserve-self-signed-certs\""
Jul 22 18:56:02.509112 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.509088    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:56:02.621152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621116    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.621152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.621444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621174    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.621444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.621444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621306    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.621444 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.621378    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxp87\" (UniqueName: \"kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722361 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722325    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxp87\" (UniqueName: \"kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722560 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722737 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722662    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.722887 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.722794    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.723033 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.723010    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.723081 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.723028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.724794 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.724763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.725045 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.725028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.730480 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.730458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxp87\" (UniqueName: \"kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87\") pod \"auth-test-model-d24283-kserve-56bcf98c5-v2qvl\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") " pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.811146 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.811054    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:02.947877 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:02.947850    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:56:02.950126 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:56:02.950097    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaadf16bb_cf39_4506_800a_a799bea44ee7.slice/crio-27945a0d235cb66c6aad2e34091651bbfee55b010a84de07e2e0a62f5285a6c9 WatchSource:0}: Error finding container 27945a0d235cb66c6aad2e34091651bbfee55b010a84de07e2e0a62f5285a6c9: Status 404 returned error can't find the container with id 27945a0d235cb66c6aad2e34091651bbfee55b010a84de07e2e0a62f5285a6c9
Jul 22 18:56:03.581780 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:03.581742    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" event={"ID":"aadf16bb-cf39-4506-800a-a799bea44ee7","Type":"ContainerStarted","Data":"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"}
Jul 22 18:56:03.581780 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:03.581787    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" event={"ID":"aadf16bb-cf39-4506-800a-a799bea44ee7","Type":"ContainerStarted","Data":"27945a0d235cb66c6aad2e34091651bbfee55b010a84de07e2e0a62f5285a6c9"}
Jul 22 18:56:03.598586 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:03.598531    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" podStartSLOduration=1.598516124 podStartE2EDuration="1.598516124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:56:03.596598904 +0000 UTC m=+2517.812246307" watchObservedRunningTime="2026-07-22 18:56:03.598516124 +0000 UTC m=+2517.814163527"
Jul 22 18:56:05.969575 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:05.969540    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:56:06.549955 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:06.549923    2580 scope.go:117] "RemoveContainer" containerID="f62b8676eb7447d7e6288e01bd46abd98c3f4d1551367835e597518f5cd373c0"
Jul 22 18:56:12.811200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:12.811147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:12.811200 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:12.811201    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:12.823543 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:12.823515    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:13.644104 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:13.644076    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:56:47.396650 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.396618    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:56:47.400163 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.400142    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.402801 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.402781    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"openshift-service-ca.crt\""
Jul 22 18:56:47.403542 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.403526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"auth-test-model-63c82d-kserve-self-signed-certs\""
Jul 22 18:56:47.403584 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.403541    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"kube-root-ca.crt\""
Jul 22 18:56:47.403738 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.403724    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"default-dockercfg-ndz89\""
Jul 22 18:56:47.408175 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.408154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:56:47.492897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.492859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h4s8\" (UniqueName: \"kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.492897 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.492904    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.493152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.492974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.493152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.493002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.493152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.493027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.493152 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.493097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594403 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h4s8\" (UniqueName: \"kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594403 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594414    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594475    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.594969 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594938    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.595048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.594945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.595048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.595017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.596724 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.596701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.597144 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.597125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.602872 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.602850    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h4s8\" (UniqueName: \"kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8\") pod \"auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") " pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.710232 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.710199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:47.846413 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:47.846391    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:56:47.848744 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:56:47.848708    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b5d54d7_c61c_4170_95af_411479ac8510.slice/crio-0c37ad748a6ba468728e584ff4ee4048caf4003b2e572e4f91109af7b0f6201f WatchSource:0}: Error finding container 0c37ad748a6ba468728e584ff4ee4048caf4003b2e572e4f91109af7b0f6201f: Status 404 returned error can't find the container with id 0c37ad748a6ba468728e584ff4ee4048caf4003b2e572e4f91109af7b0f6201f
Jul 22 18:56:48.805418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:48.805380    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" event={"ID":"6b5d54d7-c61c-4170-95af-411479ac8510","Type":"ContainerStarted","Data":"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"}
Jul 22 18:56:48.805418 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:48.805426    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" event={"ID":"6b5d54d7-c61c-4170-95af-411479ac8510","Type":"ContainerStarted","Data":"0c37ad748a6ba468728e584ff4ee4048caf4003b2e572e4f91109af7b0f6201f"}
Jul 22 18:56:48.825846 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:48.825775    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" podStartSLOduration=1.825760384 podStartE2EDuration="1.825760384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:56:48.823649133 +0000 UTC m=+2563.039296533" watchObservedRunningTime="2026-07-22 18:56:48.825760384 +0000 UTC m=+2563.041407787"
Jul 22 18:56:51.487937 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:51.487899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:56:57.710663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:57.710621    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:57.710663 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:57.710667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:57.723299 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:57.723275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:56:57.860927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:56:57.860891    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:57:04.177722 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:57:04.177679    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:57:20.677501 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:57:20.677469    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:57:38.269707 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:57:38.269627    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:57:56.836868 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:57:56.836828    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:58:14.967095 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:58:14.967061    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:58:33.377840 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:58:33.377794    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:58:48.471281 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:58:48.471245    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:58:49.405331 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:58:49.405298    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:58:52.372644 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:58:52.372609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:59:03.490917 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.490856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerName="main" containerID="cri-o://7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a" gracePeriod=45
Jul 22 18:59:03.634128 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.634082    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Jul 22 18:59:03.738248 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.738224    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:59:03.850103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850015    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850103 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850085    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxp87\" (UniqueName: \"kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850109    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850203    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir\") pod \"aadf16bb-cf39-4506-800a-a799bea44ee7\" (UID: \"aadf16bb-cf39-4506-800a-a799bea44ee7\") "
Jul 22 18:59:03.850553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache" (OuterVolumeSpecName: "model-cache") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:03.850553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home" (OuterVolumeSpecName: "home") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:03.850553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850485    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:03.850553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850504    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:03.850553 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.850525    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:03.852357 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.852327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm" (OuterVolumeSpecName: "dshm") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:03.852357 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.852342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:59:03.852501 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.852437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87" (OuterVolumeSpecName: "kube-api-access-pxp87") pod "aadf16bb-cf39-4506-800a-a799bea44ee7" (UID: "aadf16bb-cf39-4506-800a-a799bea44ee7"). InnerVolumeSpecName "kube-api-access-pxp87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:59:03.951527 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.951499    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:03.951527 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.951525    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aadf16bb-cf39-4506-800a-a799bea44ee7-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:03.951723 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.951535    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxp87\" (UniqueName: \"kubernetes.io/projected/aadf16bb-cf39-4506-800a-a799bea44ee7-kube-api-access-pxp87\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:03.951723 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:03.951547    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aadf16bb-cf39-4506-800a-a799bea44ee7-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.297872 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.297831    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 18:59:04.298314 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.298295    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerName="main"
Jul 22 18:59:04.298406 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.298316    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerName="main"
Jul 22 18:59:04.298460 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.298421    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerName="main"
Jul 22 18:59:04.301579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.301557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.304242 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.304220    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"sub-test-model-d24283-kserve-self-signed-certs\""
Jul 22 18:59:04.310196 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.310173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 18:59:04.426104 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.426051    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" podUID="6b5d54d7-c61c-4170-95af-411479ac8510" containerName="main" containerID="cri-o://6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f" gracePeriod=45
Jul 22 18:59:04.455908 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.455873    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.456031 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.455924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.456267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.456027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mppp\" (UniqueName: \"kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.456267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.456061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.456267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.456099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.456267 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.456113    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.472210 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.472179    2580 generic.go:356] "Generic (PLEG): container finished" podID="aadf16bb-cf39-4506-800a-a799bea44ee7" containerID="7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a" exitCode=0
Jul 22 18:59:04.472334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.472225    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"
Jul 22 18:59:04.472334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.472264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" event={"ID":"aadf16bb-cf39-4506-800a-a799bea44ee7","Type":"ContainerDied","Data":"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"}
Jul 22 18:59:04.472334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.472310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl" event={"ID":"aadf16bb-cf39-4506-800a-a799bea44ee7","Type":"ContainerDied","Data":"27945a0d235cb66c6aad2e34091651bbfee55b010a84de07e2e0a62f5285a6c9"}
Jul 22 18:59:04.472334 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.472327    2580 scope.go:117] "RemoveContainer" containerID="7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"
Jul 22 18:59:04.495250 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.495216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:59:04.498949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.498924    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/auth-test-model-d24283-kserve-56bcf98c5-v2qvl"]
Jul 22 18:59:04.537557 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.537427    2580 scope.go:117] "RemoveContainer" containerID="7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"
Jul 22 18:59:04.537918 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:59:04.537787    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a\": container with ID starting with 7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a not found: ID does not exist" containerID="7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"
Jul 22 18:59:04.537994 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.537936    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a"} err="failed to get container status \"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a\": rpc error: code = NotFound desc = could not find container \"7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a\": container with ID starting with 7b7065e804082aaa22db2f8ab0a058c285492602e447ecfdfd3e8b2f0970ac9a not found: ID does not exist"
Jul 22 18:59:04.556947 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.556877    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.556947 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.556913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557140 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.556957    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mppp\" (UniqueName: \"kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557140 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.556976    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557140 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.557004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557315 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.557165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557467 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.557446    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557620 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.557465    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.557686 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.557639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.559327 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.559301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.559427 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.559311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.565010 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.564989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mppp\" (UniqueName: \"kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp\") pod \"sub-test-model-d24283-kserve-68565889b5-jhh68\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") " pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.612370 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.612320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:04.670906 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.670626    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:59:04.754991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.754964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 18:59:04.756777 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:59:04.756749    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5835fdc_a17f_4bba_a6ae_75fa2045d39b.slice/crio-9cba3af9e71875df9b48189300e57523c0c07e38d092d7a81d47c77194f0e96a WatchSource:0}: Error finding container 9cba3af9e71875df9b48189300e57523c0c07e38d092d7a81d47c77194f0e96a: Status 404 returned error can't find the container with id 9cba3af9e71875df9b48189300e57523c0c07e38d092d7a81d47c77194f0e96a
Jul 22 18:59:04.758571 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758610    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h4s8\" (UniqueName: \"kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758691 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758704    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs\") pod \"6b5d54d7-c61c-4170-95af-411479ac8510\" (UID: \"6b5d54d7-c61c-4170-95af-411479ac8510\") "
Jul 22 18:59:04.758863 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758772    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home" (OuterVolumeSpecName: "home") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:04.759118 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758924    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:04.759118 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.758998    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:04.759118 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.759014    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.759118 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.759030    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.760853 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.760817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 18:59:04.760950 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.760877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm" (OuterVolumeSpecName: "dshm") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 18:59:04.760950 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.760895    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8" (OuterVolumeSpecName: "kube-api-access-9h4s8") pod "6b5d54d7-c61c-4170-95af-411479ac8510" (UID: "6b5d54d7-c61c-4170-95af-411479ac8510"). InnerVolumeSpecName "kube-api-access-9h4s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 18:59:04.860497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.860421    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.860497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.860451    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b5d54d7-c61c-4170-95af-411479ac8510-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.860497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.860459    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h4s8\" (UniqueName: \"kubernetes.io/projected/6b5d54d7-c61c-4170-95af-411479ac8510-kube-api-access-9h4s8\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:04.860497 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:04.860470    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b5d54d7-c61c-4170-95af-411479ac8510-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 18:59:05.479005 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.478975    2580 generic.go:356] "Generic (PLEG): container finished" podID="6b5d54d7-c61c-4170-95af-411479ac8510" containerID="6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f" exitCode=0
Jul 22 18:59:05.479184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.479027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"
Jul 22 18:59:05.479184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.479051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" event={"ID":"6b5d54d7-c61c-4170-95af-411479ac8510","Type":"ContainerDied","Data":"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"}
Jul 22 18:59:05.479184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.479086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv" event={"ID":"6b5d54d7-c61c-4170-95af-411479ac8510","Type":"ContainerDied","Data":"0c37ad748a6ba468728e584ff4ee4048caf4003b2e572e4f91109af7b0f6201f"}
Jul 22 18:59:05.479184 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.479110    2580 scope.go:117] "RemoveContainer" containerID="6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"
Jul 22 18:59:05.480782 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.480755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" event={"ID":"f5835fdc-a17f-4bba-a6ae-75fa2045d39b","Type":"ContainerStarted","Data":"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"}
Jul 22 18:59:05.480923 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.480793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" event={"ID":"f5835fdc-a17f-4bba-a6ae-75fa2045d39b","Type":"ContainerStarted","Data":"9cba3af9e71875df9b48189300e57523c0c07e38d092d7a81d47c77194f0e96a"}
Jul 22 18:59:05.487661 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.487643    2580 scope.go:117] "RemoveContainer" containerID="6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"
Jul 22 18:59:05.487956 ip-10-0-141-65 kubenswrapper[2580]: E0722 18:59:05.487935    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f\": container with ID starting with 6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f not found: ID does not exist" containerID="6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"
Jul 22 18:59:05.488049 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.487964    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f"} err="failed to get container status \"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f\": rpc error: code = NotFound desc = could not find container \"6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f\": container with ID starting with 6770a2d2eff4dee44b07c085b05aef19e77f6fa8c7a537f40dc2ed55e527875f not found: ID does not exist"
Jul 22 18:59:05.501727 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.501685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" podStartSLOduration=1.501667944 podStartE2EDuration="1.501667944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:59:05.499683047 +0000 UTC m=+2699.715330452" watchObservedRunningTime="2026-07-22 18:59:05.501667944 +0000 UTC m=+2699.717315348"
Jul 22 18:59:05.514593 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.514570    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:59:05.518067 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:05.518048    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/auth-test-model-63c82d-kserve-7f57fdd7c9-fc9hv"]
Jul 22 18:59:06.325949 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:06.325917    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b5d54d7-c61c-4170-95af-411479ac8510" path="/var/lib/kubelet/pods/6b5d54d7-c61c-4170-95af-411479ac8510/volumes"
Jul 22 18:59:06.326416 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:06.326399    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aadf16bb-cf39-4506-800a-a799bea44ee7" path="/var/lib/kubelet/pods/aadf16bb-cf39-4506-800a-a799bea44ee7/volumes"
Jul 22 18:59:09.369064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:09.369022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:59:14.612828 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:14.612776    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:14.613232 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:14.612846    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:14.625503 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:14.625480    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:15.542098 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:15.542064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 18:59:18.792190 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.792158    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 18:59:18.792556 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.792506    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b5d54d7-c61c-4170-95af-411479ac8510" containerName="main"
Jul 22 18:59:18.792556 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.792517    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b5d54d7-c61c-4170-95af-411479ac8510" containerName="main"
Jul 22 18:59:18.792626 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.792587    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b5d54d7-c61c-4170-95af-411479ac8510" containerName="main"
Jul 22 18:59:18.795356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.795342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.797885 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.797868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"sub-test-model-63c82d-kserve-self-signed-certs\""
Jul 22 18:59:18.797991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.797920    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"openshift-service-ca.crt\""
Jul 22 18:59:18.799105 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.799085    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"kube-root-ca.crt\""
Jul 22 18:59:18.799188 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.799159    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"default-dockercfg-ndz89\""
Jul 22 18:59:18.804751 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.804728    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 18:59:18.880991 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.880962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.881165 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.880996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.881165 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.881026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.881165 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.881115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.881165 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.881151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz6ds\" (UniqueName: \"kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.881344 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.881218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982324    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982356 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982361    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982464    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz6ds\" (UniqueName: \"kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982579 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982511    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982733    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982791 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.982943 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.982906    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.984578 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.984561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.984857 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.984836    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:18.990848 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:18.990829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz6ds\" (UniqueName: \"kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds\") pod \"sub-test-model-63c82d-kserve-8f99d4587-wmsq9\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") " pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:19.106064 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:19.106005    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:19.239173 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:19.239149    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 18:59:19.240269 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:59:19.240237    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4656b94e_fd40_4270_bf49_0d65eee2ad28.slice/crio-ed2e0c3201eef628c750754024727e7a6131b3adaed643ccd728aeaf4dd5d05e WatchSource:0}: Error finding container ed2e0c3201eef628c750754024727e7a6131b3adaed643ccd728aeaf4dd5d05e: Status 404 returned error can't find the container with id ed2e0c3201eef628c750754024727e7a6131b3adaed643ccd728aeaf4dd5d05e
Jul 22 18:59:19.554239 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:19.554205    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" event={"ID":"4656b94e-fd40-4270-bf49-0d65eee2ad28","Type":"ContainerStarted","Data":"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"}
Jul 22 18:59:19.554239 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:19.554245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" event={"ID":"4656b94e-fd40-4270-bf49-0d65eee2ad28","Type":"ContainerStarted","Data":"ed2e0c3201eef628c750754024727e7a6131b3adaed643ccd728aeaf4dd5d05e"}
Jul 22 18:59:22.772890 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:22.772840    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" podStartSLOduration=4.7728248220000005 podStartE2EDuration="4.772824822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:59:19.591240972 +0000 UTC m=+2713.806888387" watchObservedRunningTime="2026-07-22 18:59:22.772824822 +0000 UTC m=+2716.988472309"
Jul 22 18:59:22.773915 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:22.773898    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:59:29.106927 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:29.106897    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:29.109374 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:29.106961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:29.119607 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:29.119583    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:29.614320 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:29.614291    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 18:59:36.471544 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:36.471511    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 18:59:52.901434 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:52.901398    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 18:59:53.770894 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:53.770861    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 18:59:57.896903 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:57.896861    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 18:59:57.900327 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:57.900307    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:57.902888 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:57.902867    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24283\"/\"rate-test-model-d24283-kserve-self-signed-certs\""
Jul 22 18:59:57.918127 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:57.918101    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 18:59:58.005875 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.005831    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.006065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.005887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2pt2\" (UniqueName: \"kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.006065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.005950    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.006065 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.006038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.006237 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.006080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.006237 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.006135    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107109 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107073    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107311 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2pt2\" (UniqueName: \"kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107311 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107311 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107176    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107311 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107195    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107311 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107228    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107575 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107644 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107589    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.107697 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.107631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.109392 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.109373    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.109699 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.109681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.115625 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.115600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2pt2\" (UniqueName: \"kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2\") pod \"rate-test-model-d24283-kserve-69969c7899-65wkp\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") " pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.210974 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.210943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 18:59:58.350161 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.350140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 18:59:58.352291 ip-10-0-141-65 kubenswrapper[2580]: W0722 18:59:58.352263    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb40e727f_a3c6_4f10_9b68_1aa75be45139.slice/crio-4f2f1791d11e4e8034c5aa2062347904daf8502a293a3381400bbfb6d558517f WatchSource:0}: Error finding container 4f2f1791d11e4e8034c5aa2062347904daf8502a293a3381400bbfb6d558517f: Status 404 returned error can't find the container with id 4f2f1791d11e4e8034c5aa2062347904daf8502a293a3381400bbfb6d558517f
Jul 22 18:59:58.755512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.755471    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" event={"ID":"b40e727f-a3c6-4f10-9b68-1aa75be45139","Type":"ContainerStarted","Data":"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"}
Jul 22 18:59:58.755512 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.755513    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" event={"ID":"b40e727f-a3c6-4f10-9b68-1aa75be45139","Type":"ContainerStarted","Data":"4f2f1791d11e4e8034c5aa2062347904daf8502a293a3381400bbfb6d558517f"}
Jul 22 18:59:58.777204 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.777152    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" podStartSLOduration=1.777138845 podStartE2EDuration="1.777138845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 18:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 18:59:58.775232229 +0000 UTC m=+2752.990879632" watchObservedRunningTime="2026-07-22 18:59:58.777138845 +0000 UTC m=+2752.992786249"
Jul 22 18:59:58.889048 ip-10-0-141-65 kubenswrapper[2580]: I0722 18:59:58.889011    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:00:06.600360 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:06.600327    2580 scope.go:117] "RemoveContainer" containerID="cc71bd0a8ba982b39e22a01e97f90d5d503f5089006fd9d8dcdc67c64482ac8a"
Jul 22 19:00:07.921100 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:07.921036    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" podUID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" containerName="main" containerID="cri-o://0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d" gracePeriod=45
Jul 22 19:00:08.170880 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.170856    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 19:00:08.211097 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.211068    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:00:08.211286 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.211112    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:00:08.223399 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.223377    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:00:08.296592 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.296784 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296617    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.296784 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mppp\" (UniqueName: \"kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.296784 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.296784 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296691    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.297053 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296870    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home" (OuterVolumeSpecName: "home") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:08.297053 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.296904    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm\") pod \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\" (UID: \"f5835fdc-a17f-4bba-a6ae-75fa2045d39b\") "
Jul 22 19:00:08.297053 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.297009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:08.297053 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.297020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:08.297221 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.297214    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.297277 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.297226    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.297277 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.297239    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.298823 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.298791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:00:08.298893 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.298829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm" (OuterVolumeSpecName: "dshm") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:08.298893 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.298833    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp" (OuterVolumeSpecName: "kube-api-access-9mppp") pod "f5835fdc-a17f-4bba-a6ae-75fa2045d39b" (UID: "f5835fdc-a17f-4bba-a6ae-75fa2045d39b"). InnerVolumeSpecName "kube-api-access-9mppp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:00:08.398423 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.398394    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.398423 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.398418    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mppp\" (UniqueName: \"kubernetes.io/projected/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-kube-api-access-9mppp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.398423 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.398427    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5835fdc-a17f-4bba-a6ae-75fa2045d39b-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:08.790554 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.790486    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" podUID="4656b94e-fd40-4270-bf49-0d65eee2ad28" containerName="main" containerID="cri-o://ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a" gracePeriod=45
Jul 22 19:00:08.803584 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.803559    2580 generic.go:356] "Generic (PLEG): container finished" podID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" containerID="0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d" exitCode=0
Jul 22 19:00:08.803726 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.803607    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"
Jul 22 19:00:08.803726 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.803645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" event={"ID":"f5835fdc-a17f-4bba-a6ae-75fa2045d39b","Type":"ContainerDied","Data":"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"}
Jul 22 19:00:08.803726 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.803694    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68" event={"ID":"f5835fdc-a17f-4bba-a6ae-75fa2045d39b","Type":"ContainerDied","Data":"9cba3af9e71875df9b48189300e57523c0c07e38d092d7a81d47c77194f0e96a"}
Jul 22 19:00:08.803726 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.803715    2580 scope.go:117] "RemoveContainer" containerID="0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"
Jul 22 19:00:08.815100 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.815074    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:00:08.873441 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.873411    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 19:00:08.879485 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.879460    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/sub-test-model-d24283-kserve-68565889b5-jhh68"]
Jul 22 19:00:08.901753 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.901731    2580 scope.go:117] "RemoveContainer" containerID="0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"
Jul 22 19:00:08.902128 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:00:08.902107    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d\": container with ID starting with 0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d not found: ID does not exist" containerID="0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"
Jul 22 19:00:08.902203 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:08.902136    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d"} err="failed to get container status \"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d\": rpc error: code = NotFound desc = could not find container \"0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d\": container with ID starting with 0ef1b684fca819f72fb3fdef819c9a85ffed3ab095568d0504d0a2045dd6c88d not found: ID does not exist"
Jul 22 19:00:09.052533 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.052477    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 19:00:09.205785 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205740    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.205980 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205847    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.205980 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205878    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.205980 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205914    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz6ds\" (UniqueName: \"kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.205980 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205934    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.205986    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home\") pod \"4656b94e-fd40-4270-bf49-0d65eee2ad28\" (UID: \"4656b94e-fd40-4270-bf49-0d65eee2ad28\") "
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.206087    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.206225    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache" (OuterVolumeSpecName: "model-cache") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.206292    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home" (OuterVolumeSpecName: "home") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.206309    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.206482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.206328    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.208043 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.208016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds" (OuterVolumeSpecName: "kube-api-access-xz6ds") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "kube-api-access-xz6ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:00:09.208449 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.208424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm" (OuterVolumeSpecName: "dshm") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:00:09.208547 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.208443    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4656b94e-fd40-4270-bf49-0d65eee2ad28" (UID: "4656b94e-fd40-4270-bf49-0d65eee2ad28"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:00:09.270683 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.270644    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:00:09.307516 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.307449    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.307516 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.307470    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4656b94e-fd40-4270-bf49-0d65eee2ad28-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.307516 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.307480    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4656b94e-fd40-4270-bf49-0d65eee2ad28-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.307516 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.307490    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz6ds\" (UniqueName: \"kubernetes.io/projected/4656b94e-fd40-4270-bf49-0d65eee2ad28-kube-api-access-xz6ds\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:00:09.816297 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.816258    2580 generic.go:356] "Generic (PLEG): container finished" podID="4656b94e-fd40-4270-bf49-0d65eee2ad28" containerID="ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a" exitCode=0
Jul 22 19:00:09.816492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.816316    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"
Jul 22 19:00:09.816492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.816343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" event={"ID":"4656b94e-fd40-4270-bf49-0d65eee2ad28","Type":"ContainerDied","Data":"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"}
Jul 22 19:00:09.816492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.816386    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9" event={"ID":"4656b94e-fd40-4270-bf49-0d65eee2ad28","Type":"ContainerDied","Data":"ed2e0c3201eef628c750754024727e7a6131b3adaed643ccd728aeaf4dd5d05e"}
Jul 22 19:00:09.816492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.816403    2580 scope.go:117] "RemoveContainer" containerID="ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"
Jul 22 19:00:09.825377 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.825361    2580 scope.go:117] "RemoveContainer" containerID="ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"
Jul 22 19:00:09.825630 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:00:09.825612    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a\": container with ID starting with ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a not found: ID does not exist" containerID="ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"
Jul 22 19:00:09.825696 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.825640    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a"} err="failed to get container status \"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a\": rpc error: code = NotFound desc = could not find container \"ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a\": container with ID starting with ebe24f8db1f63bddb805cb36bd05b99fed09ff995fd613ac6dc04a62a2aab64a not found: ID does not exist"
Jul 22 19:00:09.838174 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.838142    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 19:00:09.842091 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:09.842070    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/sub-test-model-63c82d-kserve-8f99d4587-wmsq9"]
Jul 22 19:00:10.320953 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:10.320921    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4656b94e-fd40-4270-bf49-0d65eee2ad28" path="/var/lib/kubelet/pods/4656b94e-fd40-4270-bf49-0d65eee2ad28/volumes"
Jul 22 19:00:10.321317 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:10.321279    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" path="/var/lib/kubelet/pods/f5835fdc-a17f-4bba-a6ae-75fa2045d39b/volumes"
Jul 22 19:00:11.695417 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695380    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:00:11.695832 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695755    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" containerName="main"
Jul 22 19:00:11.695832 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695766    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" containerName="main"
Jul 22 19:00:11.695832 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695782    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4656b94e-fd40-4270-bf49-0d65eee2ad28" containerName="main"
Jul 22 19:00:11.695832 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695787    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4656b94e-fd40-4270-bf49-0d65eee2ad28" containerName="main"
Jul 22 19:00:11.696152 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695870    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4656b94e-fd40-4270-bf49-0d65eee2ad28" containerName="main"
Jul 22 19:00:11.696152 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.695883    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5835fdc-a17f-4bba-a6ae-75fa2045d39b" containerName="main"
Jul 22 19:00:11.700525 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.700334    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.703787 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.703764    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"rate-test-model-63c82d-kserve-self-signed-certs\""
Jul 22 19:00:11.703935 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.703779    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"default-dockercfg-ndz89\""
Jul 22 19:00:11.703935 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.703852    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"kube-root-ca.crt\""
Jul 22 19:00:11.703935 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.703786    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-63c82d\"/\"openshift-service-ca.crt\""
Jul 22 19:00:11.710594 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.710566    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:00:11.733562 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.733693 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.733693 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733614    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.733835 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.733835 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.733940 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.733853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpdb\" (UniqueName: \"kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834537 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834713 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpdb\" (UniqueName: \"kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834713 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834617    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834713 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834713 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834959 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.834959 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.834931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.835075 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.835018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.835151 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.835129    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.836912 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.836890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.837184 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.837169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:11.843153 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:11.843134    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpdb\" (UniqueName: \"kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb\") pod \"rate-test-model-63c82d-kserve-5466749499-zfczm\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") " pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:12.011410 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:12.011316    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:12.146624 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:12.146544    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:00:12.149293 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:00:12.149266    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda79fe3ab_c074_4562_8463_a7efde34db97.slice/crio-18f0accd03d617cd11f94f7814cb23b5ae99450d81217a1801fd2781f9081de4 WatchSource:0}: Error finding container 18f0accd03d617cd11f94f7814cb23b5ae99450d81217a1801fd2781f9081de4: Status 404 returned error can't find the container with id 18f0accd03d617cd11f94f7814cb23b5ae99450d81217a1801fd2781f9081de4
Jul 22 19:00:12.836620 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:12.836576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" event={"ID":"a79fe3ab-c074-4562-8463-a7efde34db97","Type":"ContainerStarted","Data":"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"}
Jul 22 19:00:12.836620 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:12.836625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" event={"ID":"a79fe3ab-c074-4562-8463-a7efde34db97","Type":"ContainerStarted","Data":"18f0accd03d617cd11f94f7814cb23b5ae99450d81217a1801fd2781f9081de4"}
Jul 22 19:00:12.854565 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:12.854517    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" podStartSLOduration=1.8545020829999999 podStartE2EDuration="1.854502083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:00:12.85296864 +0000 UTC m=+2767.068616046" watchObservedRunningTime="2026-07-22 19:00:12.854502083 +0000 UTC m=+2767.070149486"
Jul 22 19:00:21.596825 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:21.596781    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:00:22.011678 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:22.011647    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:22.011865 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:22.011725    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:22.024259 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:22.024228    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:22.896132 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:22.896107    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:00:43.172859 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:43.172756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:00:57.877997 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:00:57.877958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:01:04.778005 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:04.777972    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 19:01:05.621430 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:05.621397    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:01:06.619957 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:06.619923    2580 scope.go:117] "RemoveContainer" containerID="89a2e7bfbce527c4eec299350e1e5617f7cb4944e957c8a33a6b4588573e03c0"
Jul 22 19:01:06.628000 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:06.627978    2580 scope.go:117] "RemoveContainer" containerID="7401efe84dd40322f8f2e2cfaf5237f9edc5d66f67c5fe4482e6e9758784e105"
Jul 22 19:01:18.269578 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:18.269545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:01:19.797798 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:19.797737    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" podUID="b40e727f-a3c6-4f10-9b68-1aa75be45139" containerName="main" containerID="cri-o://f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6" gracePeriod=45
Jul 22 19:01:20.051236 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.051182    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:01:20.115384 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115358    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2pt2\" (UniqueName: \"kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115512 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115401    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115512 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115417    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115512 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115455    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115512 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115491    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115733 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115514    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir\") pod \"b40e727f-a3c6-4f10-9b68-1aa75be45139\" (UID: \"b40e727f-a3c6-4f10-9b68-1aa75be45139\") "
Jul 22 19:01:20.115793 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache" (OuterVolumeSpecName: "model-cache") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.115886 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115839    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home" (OuterVolumeSpecName: "home") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.115886 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115877    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.115966 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.115898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.117525 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.117497    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm" (OuterVolumeSpecName: "dshm") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.117525 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.117510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2" (OuterVolumeSpecName: "kube-api-access-h2pt2") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "kube-api-access-h2pt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:01:20.117655 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.117551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b40e727f-a3c6-4f10-9b68-1aa75be45139" (UID: "b40e727f-a3c6-4f10-9b68-1aa75be45139"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:01:20.166441 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.166414    2580 generic.go:356] "Generic (PLEG): container finished" podID="b40e727f-a3c6-4f10-9b68-1aa75be45139" containerID="f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6" exitCode=0
Jul 22 19:01:20.166563 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.166459    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"
Jul 22 19:01:20.166563 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.166507    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" event={"ID":"b40e727f-a3c6-4f10-9b68-1aa75be45139","Type":"ContainerDied","Data":"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"}
Jul 22 19:01:20.166563 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.166555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp" event={"ID":"b40e727f-a3c6-4f10-9b68-1aa75be45139","Type":"ContainerDied","Data":"4f2f1791d11e4e8034c5aa2062347904daf8502a293a3381400bbfb6d558517f"}
Jul 22 19:01:20.166732 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.166578    2580 scope.go:117] "RemoveContainer" containerID="f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"
Jul 22 19:01:20.175035 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.175017    2580 scope.go:117] "RemoveContainer" containerID="f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"
Jul 22 19:01:20.175267 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:01:20.175249    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6\": container with ID starting with f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6 not found: ID does not exist" containerID="f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"
Jul 22 19:01:20.175316 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.175276    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6"} err="failed to get container status \"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6\": rpc error: code = NotFound desc = could not find container \"f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6\": container with ID starting with f0f9bee1e644def035232be35cbf9425874f77542f3f161294270e55676962e6 not found: ID does not exist"
Jul 22 19:01:20.187141 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.187122    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 19:01:20.191234 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.191216    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24283/rate-test-model-d24283-kserve-69969c7899-65wkp"]
Jul 22 19:01:20.216898 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.216876    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2pt2\" (UniqueName: \"kubernetes.io/projected/b40e727f-a3c6-4f10-9b68-1aa75be45139-kube-api-access-h2pt2\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.216961 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.216898    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.216961 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.216908    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.216961 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.216916    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40e727f-a3c6-4f10-9b68-1aa75be45139-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.216961 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.216925    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b40e727f-a3c6-4f10-9b68-1aa75be45139-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:20.320306 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.320253    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b40e727f-a3c6-4f10-9b68-1aa75be45139" path="/var/lib/kubelet/pods/b40e727f-a3c6-4f10-9b68-1aa75be45139/volumes"
Jul 22 19:01:20.641235 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.641140    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" podUID="a79fe3ab-c074-4562-8463-a7efde34db97" containerName="main" containerID="cri-o://a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2" gracePeriod=45
Jul 22 19:01:20.890889 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.890865    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:01:20.922989 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.922933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.922989 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.922968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.923167 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923014    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.923167 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923029    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.923167 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923056    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.923167 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923099    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lpdb\" (UniqueName: \"kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb\") pod \"a79fe3ab-c074-4562-8463-a7efde34db97\" (UID: \"a79fe3ab-c074-4562-8463-a7efde34db97\") "
Jul 22 19:01:20.923369 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home" (OuterVolumeSpecName: "home") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.923369 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923309    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.923369 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.923329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache" (OuterVolumeSpecName: "model-cache") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.925014 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.924993    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm" (OuterVolumeSpecName: "dshm") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 22 19:01:20.925206 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.925187    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:01:20.925206 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:20.925192    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb" (OuterVolumeSpecName: "kube-api-access-6lpdb") pod "a79fe3ab-c074-4562-8463-a7efde34db97" (UID: "a79fe3ab-c074-4562-8463-a7efde34db97"). InnerVolumeSpecName "kube-api-access-6lpdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:01:21.024206 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024182    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-home\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.024206 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024202    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-model-cache\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.024206 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024212    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79fe3ab-c074-4562-8463-a7efde34db97-tls-certs\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.024375 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024220    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpdb\" (UniqueName: \"kubernetes.io/projected/a79fe3ab-c074-4562-8463-a7efde34db97-kube-api-access-6lpdb\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.024375 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024228    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-dshm\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.024375 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.024236    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79fe3ab-c074-4562-8463-a7efde34db97-tmp-dir\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:21.172777 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.172743    2580 generic.go:356] "Generic (PLEG): container finished" podID="a79fe3ab-c074-4562-8463-a7efde34db97" containerID="a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2" exitCode=0
Jul 22 19:01:21.172924 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.172789    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"
Jul 22 19:01:21.172924 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.172833    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" event={"ID":"a79fe3ab-c074-4562-8463-a7efde34db97","Type":"ContainerDied","Data":"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"}
Jul 22 19:01:21.172924 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.172868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm" event={"ID":"a79fe3ab-c074-4562-8463-a7efde34db97","Type":"ContainerDied","Data":"18f0accd03d617cd11f94f7814cb23b5ae99450d81217a1801fd2781f9081de4"}
Jul 22 19:01:21.172924 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.172885    2580 scope.go:117] "RemoveContainer" containerID="a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"
Jul 22 19:01:21.181915 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.181895    2580 scope.go:117] "RemoveContainer" containerID="a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"
Jul 22 19:01:21.182141 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:01:21.182121    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2\": container with ID starting with a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2 not found: ID does not exist" containerID="a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"
Jul 22 19:01:21.182202 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.182148    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2"} err="failed to get container status \"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2\": rpc error: code = NotFound desc = could not find container \"a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2\": container with ID starting with a98c680967dedd3b7416e0335ee54fa2c05f19c9669503c5356503deb513f6f2 not found: ID does not exist"
Jul 22 19:01:21.194125 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.194104    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:01:21.198697 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.198677    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-63c82d/rate-test-model-63c82d-kserve-5466749499-zfczm"]
Jul 22 19:01:21.973840 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.973787    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7497-139754714880016"]
Jul 22 19:01:21.974339 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974321    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79fe3ab-c074-4562-8463-a7efde34db97" containerName="main"
Jul 22 19:01:21.974417 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974342    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79fe3ab-c074-4562-8463-a7efde34db97" containerName="main"
Jul 22 19:01:21.974417 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974373    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b40e727f-a3c6-4f10-9b68-1aa75be45139" containerName="main"
Jul 22 19:01:21.974417 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974382    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b40e727f-a3c6-4f10-9b68-1aa75be45139" containerName="main"
Jul 22 19:01:21.974564 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974475    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b40e727f-a3c6-4f10-9b68-1aa75be45139" containerName="main"
Jul 22 19:01:21.974564 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.974491    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79fe3ab-c074-4562-8463-a7efde34db97" containerName="main"
Jul 22 19:01:21.978886 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.978865    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:21.982720 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.982700    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pjbvh\""
Jul 22 19:01:21.984452 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:21.984433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-139754714880016"]
Jul 22 19:01:22.032346 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.032325    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8dpw\" (UniqueName: \"kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw\") pod \"test-curl-7497-139754714880016\" (UID: \"50109244-28d9-4535-b194-20d163fde388\") " pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:22.133373 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.133348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8dpw\" (UniqueName: \"kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw\") pod \"test-curl-7497-139754714880016\" (UID: \"50109244-28d9-4535-b194-20d163fde388\") " pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:22.142854 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.142828    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8dpw\" (UniqueName: \"kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw\") pod \"test-curl-7497-139754714880016\" (UID: \"50109244-28d9-4535-b194-20d163fde388\") " pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:22.288992 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.288929    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:22.321696 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.321664    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79fe3ab-c074-4562-8463-a7efde34db97" path="/var/lib/kubelet/pods/a79fe3ab-c074-4562-8463-a7efde34db97/volumes"
Jul 22 19:01:22.433915 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.433826    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7497-139754714880016"]
Jul 22 19:01:22.436759 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:01:22.436729    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50109244_28d9_4535_b194_20d163fde388.slice/crio-dec588dc0d49b1eb2d5d1795e302612d9b9226784e417952fa113fa887438b4f WatchSource:0}: Error finding container dec588dc0d49b1eb2d5d1795e302612d9b9226784e417952fa113fa887438b4f: Status 404 returned error can't find the container with id dec588dc0d49b1eb2d5d1795e302612d9b9226784e417952fa113fa887438b4f
Jul 22 19:01:22.437960 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:22.437940    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 22 19:01:23.185244 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:23.185220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-139754714880016" event={"ID":"50109244-28d9-4535-b194-20d163fde388","Type":"ContainerStarted","Data":"dec588dc0d49b1eb2d5d1795e302612d9b9226784e417952fa113fa887438b4f"}
Jul 22 19:01:24.191468 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:24.191431    2580 generic.go:356] "Generic (PLEG): container finished" podID="50109244-28d9-4535-b194-20d163fde388" containerID="90b015027bbfd7c95b12dd37c72ee90c5fca09abdda615a1003f498e909927ad" exitCode=0
Jul 22 19:01:24.191866 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:24.191515    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7497-139754714880016" event={"ID":"50109244-28d9-4535-b194-20d163fde388","Type":"ContainerDied","Data":"90b015027bbfd7c95b12dd37c72ee90c5fca09abdda615a1003f498e909927ad"}
Jul 22 19:01:25.328291 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.328269    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:25.383773 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.383746    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7497-139754714880016_50109244-28d9-4535-b194-20d163fde388/test-curl-7497-139754714880016/0.log"
Jul 22 19:01:25.424475 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.424448    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7497-139754714880016"]
Jul 22 19:01:25.426377 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.426357    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7497-139754714880016"]
Jul 22 19:01:25.462058 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.461994    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8dpw\" (UniqueName: \"kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw\") pod \"50109244-28d9-4535-b194-20d163fde388\" (UID: \"50109244-28d9-4535-b194-20d163fde388\") "
Jul 22 19:01:25.463947 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.463921    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw" (OuterVolumeSpecName: "kube-api-access-q8dpw") pod "50109244-28d9-4535-b194-20d163fde388" (UID: "50109244-28d9-4535-b194-20d163fde388"). InnerVolumeSpecName "kube-api-access-q8dpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:01:25.563049 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.563027    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8dpw\" (UniqueName: \"kubernetes.io/projected/50109244-28d9-4535-b194-20d163fde388-kube-api-access-q8dpw\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:25.649418 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.649391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"]
Jul 22 19:01:25.649838 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.649820    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50109244-28d9-4535-b194-20d163fde388" containerName="test-curl-7497-139754714880016"
Jul 22 19:01:25.649929 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.649840    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="50109244-28d9-4535-b194-20d163fde388" containerName="test-curl-7497-139754714880016"
Jul 22 19:01:25.649983 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.649975    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="50109244-28d9-4535-b194-20d163fde388" containerName="test-curl-7497-139754714880016"
Jul 22 19:01:25.653220 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.653202    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:25.656532 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.656515    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 19:01:25.664368 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.664346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"]
Jul 22 19:01:25.764123 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.764059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24283-844vv\" (UID: \"7ab692aa-58c5-4b46-ba3e-785339ba8898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:25.864482 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.864457    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24283-844vv\" (UID: \"7ab692aa-58c5-4b46-ba3e-785339ba8898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:25.865043 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.865025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d24283-844vv\" (UID: \"7ab692aa-58c5-4b46-ba3e-785339ba8898\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:25.962881 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:25.962857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:26.112492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:26.112348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"]
Jul 22 19:01:26.115033 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:01:26.115011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab692aa_58c5_4b46_ba3e_785339ba8898.slice/crio-09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b WatchSource:0}: Error finding container 09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b: Status 404 returned error can't find the container with id 09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b
Jul 22 19:01:26.202051 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:26.202033    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:26.202175 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:26.202076    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec588dc0d49b1eb2d5d1795e302612d9b9226784e417952fa113fa887438b4f"
Jul 22 19:01:26.203343 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:26.203320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv" event={"ID":"7ab692aa-58c5-4b46-ba3e-785339ba8898","Type":"ContainerStarted","Data":"09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b"}
Jul 22 19:01:26.212061 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:01:26.212024    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-139754714880016\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="50109244-28d9-4535-b194-20d163fde388" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:26.319463 ip-10-0-141-65 kubenswrapper[2580]: E0722 19:01:26.319407    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7497-139754714880016\" is forbidden: User \"system:node:ip-10-0-141-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-65.ec2.internal' and this object" podUID="50109244-28d9-4535-b194-20d163fde388" pod="opendatahub/test-curl-7497-139754714880016"
Jul 22 19:01:26.320673 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:26.320654    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50109244-28d9-4535-b194-20d163fde388" path="/var/lib/kubelet/pods/50109244-28d9-4535-b194-20d163fde388/volumes"
Jul 22 19:01:27.210043 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:27.210013    2580 generic.go:356] "Generic (PLEG): container finished" podID="7ab692aa-58c5-4b46-ba3e-785339ba8898" containerID="42df763a64cbf7280629b15a62a0acd67346d43e9f9aa91f0bb62477c39db0f8" exitCode=0
Jul 22 19:01:27.210416 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:27.210106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv" event={"ID":"7ab692aa-58c5-4b46-ba3e-785339ba8898","Type":"ContainerDied","Data":"42df763a64cbf7280629b15a62a0acd67346d43e9f9aa91f0bb62477c39db0f8"}
Jul 22 19:01:28.347091 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:28.347070    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:28.487630 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:28.487561    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca\") pod \"7ab692aa-58c5-4b46-ba3e-785339ba8898\" (UID: \"7ab692aa-58c5-4b46-ba3e-785339ba8898\") "
Jul 22 19:01:28.487959 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:28.487931    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ab692aa-58c5-4b46-ba3e-785339ba8898" (UID: "7ab692aa-58c5-4b46-ba3e-785339ba8898"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 19:01:28.588999 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:28.588973    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab692aa-58c5-4b46-ba3e-785339ba8898-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:29.221025 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:29.220990    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"
Jul 22 19:01:29.221025 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:29.221020    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv" event={"ID":"7ab692aa-58c5-4b46-ba3e-785339ba8898","Type":"ContainerDied","Data":"09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b"}
Jul 22 19:01:29.221261 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:29.221080    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09dad515410ea7240144022b35713f28b0971fff671f1772ed5273901713cf8b"
Jul 22 19:01:36.057964 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.057932    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 19:01:36.058385 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.058125    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" podUID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" containerName="maas-api" containerID="cri-o://125e154c215e811055a7f5fd241885f0636d3f11c3488eba6e0d42bdd2ec842d" gracePeriod=30
Jul 22 19:01:36.254491 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.254460    2580 generic.go:356] "Generic (PLEG): container finished" podID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" containerID="125e154c215e811055a7f5fd241885f0636d3f11c3488eba6e0d42bdd2ec842d" exitCode=0
Jul 22 19:01:36.254649 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.254531    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" event={"ID":"df8cbadd-c040-446d-bf72-12b7a3c5cfe7","Type":"ContainerDied","Data":"125e154c215e811055a7f5fd241885f0636d3f11c3488eba6e0d42bdd2ec842d"}
Jul 22 19:01:36.306440 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.306419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 19:01:36.452248 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.452223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq7lp\" (UniqueName: \"kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp\") pod \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") "
Jul 22 19:01:36.452389 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.452280    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca\") pod \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") "
Jul 22 19:01:36.452389 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.452315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls\") pod \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\" (UID: \"df8cbadd-c040-446d-bf72-12b7a3c5cfe7\") "
Jul 22 19:01:36.452640 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.452609    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df8cbadd-c040-446d-bf72-12b7a3c5cfe7" (UID: "df8cbadd-c040-446d-bf72-12b7a3c5cfe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 19:01:36.454274 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.454247    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp" (OuterVolumeSpecName: "kube-api-access-fq7lp") pod "df8cbadd-c040-446d-bf72-12b7a3c5cfe7" (UID: "df8cbadd-c040-446d-bf72-12b7a3c5cfe7"). InnerVolumeSpecName "kube-api-access-fq7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:01:36.454363 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.454337    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df8cbadd-c040-446d-bf72-12b7a3c5cfe7" (UID: "df8cbadd-c040-446d-bf72-12b7a3c5cfe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:01:36.552999 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.552971    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq7lp\" (UniqueName: \"kubernetes.io/projected/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-kube-api-access-fq7lp\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:36.552999 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.552995    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:36.553167 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:36.553005    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8cbadd-c040-446d-bf72-12b7a3c5cfe7-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:37.260763 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:37.260731    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw" event={"ID":"df8cbadd-c040-446d-bf72-12b7a3c5cfe7","Type":"ContainerDied","Data":"0303e1bea438fd92c49a3f463d667ce5c56c22ea2f7a79e440a25591a6f102db"}
Jul 22 19:01:37.261196 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:37.260739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"
Jul 22 19:01:37.261196 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:37.260798    2580 scope.go:117] "RemoveContainer" containerID="125e154c215e811055a7f5fd241885f0636d3f11c3488eba6e0d42bdd2ec842d"
Jul 22 19:01:37.283963 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:37.283943    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 19:01:37.287868 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:37.287848    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-d24283-788454d79d-287hw"]
Jul 22 19:01:38.320686 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:38.320651    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" path="/var/lib/kubelet/pods/df8cbadd-c040-446d-bf72-12b7a3c5cfe7/volumes"
Jul 22 19:01:39.177376 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:39.177344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:01:40.726336 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:40.726297    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"]
Jul 22 19:01:40.728674 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:40.728647    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d24283-844vv"]
Jul 22 19:01:42.320142 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:42.320100    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab692aa-58c5-4b46-ba3e-785339ba8898" path="/var/lib/kubelet/pods/7ab692aa-58c5-4b46-ba3e-785339ba8898/volumes"
Jul 22 19:01:48.724723 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.724687    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"]
Jul 22 19:01:48.725159 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725076    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" containerName="maas-api"
Jul 22 19:01:48.725159 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725088    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" containerName="maas-api"
Jul 22 19:01:48.725159 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725114    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab692aa-58c5-4b46-ba3e-785339ba8898" containerName="revoke-keys"
Jul 22 19:01:48.725159 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725120    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab692aa-58c5-4b46-ba3e-785339ba8898" containerName="revoke-keys"
Jul 22 19:01:48.725295 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725174    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="df8cbadd-c040-446d-bf72-12b7a3c5cfe7" containerName="maas-api"
Jul 22 19:01:48.725295 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.725186    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab692aa-58c5-4b46-ba3e-785339ba8898" containerName="revoke-keys"
Jul 22 19:01:48.734255 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.734228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:48.737801 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.737771    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7279\""
Jul 22 19:01:48.739675 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.739650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"]
Jul 22 19:01:48.854181 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.854148    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt\" (UID: \"76596da6-4932-4939-896b-0993d7b8cee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:48.955068 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.955041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt\" (UID: \"76596da6-4932-4939-896b-0993d7b8cee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:48.955682 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:48.955663    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt\" (UID: \"76596da6-4932-4939-896b-0993d7b8cee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:49.046577 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:49.046503    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:49.192869 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:49.192833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"]
Jul 22 19:01:49.196086 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:01:49.196055    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76596da6_4932_4939_896b_0993d7b8cee1.slice/crio-53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594 WatchSource:0}: Error finding container 53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594: Status 404 returned error can't find the container with id 53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594
Jul 22 19:01:49.324033 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:49.323946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt" event={"ID":"76596da6-4932-4939-896b-0993d7b8cee1","Type":"ContainerStarted","Data":"53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594"}
Jul 22 19:01:50.329500 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:50.329468    2580 generic.go:356] "Generic (PLEG): container finished" podID="76596da6-4932-4939-896b-0993d7b8cee1" containerID="14544c99d33910ede78abe47424ff8592c8bf9e7a0e6364f3ca226cc54a084fe" exitCode=0
Jul 22 19:01:50.329919 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:50.329555    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt" event={"ID":"76596da6-4932-4939-896b-0993d7b8cee1","Type":"ContainerDied","Data":"14544c99d33910ede78abe47424ff8592c8bf9e7a0e6364f3ca226cc54a084fe"}
Jul 22 19:01:51.467121 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:51.467095    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:51.578781 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:51.578744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca\") pod \"76596da6-4932-4939-896b-0993d7b8cee1\" (UID: \"76596da6-4932-4939-896b-0993d7b8cee1\") "
Jul 22 19:01:51.579135 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:51.579110    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "76596da6-4932-4939-896b-0993d7b8cee1" (UID: "76596da6-4932-4939-896b-0993d7b8cee1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 19:01:51.679366 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:51.679333    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76596da6-4932-4939-896b-0993d7b8cee1-maas-api-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:52.340838 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:52.340796    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"
Jul 22 19:01:52.340976 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:52.340841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt" event={"ID":"76596da6-4932-4939-896b-0993d7b8cee1","Type":"ContainerDied","Data":"53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594"}
Jul 22 19:01:52.340976 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:52.340887    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53960583a09f7223b685c5dde8af9ff0647d37fd1f9d3dcc492127b3ef000594"
Jul 22 19:01:59.154862 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.154798    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 19:01:59.155319 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.155068    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" podUID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" containerName="maas-api" containerID="cri-o://ba5a241944de83f357c13852a22e0d75c302739841266ae6cd4d177771312dd6" gracePeriod=30
Jul 22 19:01:59.376284 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.376251    2580 generic.go:356] "Generic (PLEG): container finished" podID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" containerID="ba5a241944de83f357c13852a22e0d75c302739841266ae6cd4d177771312dd6" exitCode=0
Jul 22 19:01:59.376436 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.376320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" event={"ID":"111e1044-6683-459f-9e1f-3e0cb9b8fd4c","Type":"ContainerDied","Data":"ba5a241944de83f357c13852a22e0d75c302739841266ae6cd4d177771312dd6"}
Jul 22 19:01:59.398842 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.398790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 19:01:59.443726 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.443699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvfv\" (UniqueName: \"kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv\") pod \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") "
Jul 22 19:01:59.443894 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.443744    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") pod \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") "
Jul 22 19:01:59.443894 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.443826    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca\") pod \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\" (UID: \"111e1044-6683-459f-9e1f-3e0cb9b8fd4c\") "
Jul 22 19:01:59.444226 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.444202    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "111e1044-6683-459f-9e1f-3e0cb9b8fd4c" (UID: "111e1044-6683-459f-9e1f-3e0cb9b8fd4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 22 19:01:59.445864 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.445840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv" (OuterVolumeSpecName: "kube-api-access-znvfv") pod "111e1044-6683-459f-9e1f-3e0cb9b8fd4c" (UID: "111e1044-6683-459f-9e1f-3e0cb9b8fd4c"). InnerVolumeSpecName "kube-api-access-znvfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 22 19:01:59.445918 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.445859    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "111e1044-6683-459f-9e1f-3e0cb9b8fd4c" (UID: "111e1044-6683-459f-9e1f-3e0cb9b8fd4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 22 19:01:59.544641 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.544611    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-openshift-service-ca\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:59.544641 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.544639    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvfv\" (UniqueName: \"kubernetes.io/projected/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-kube-api-access-znvfv\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:01:59.544641 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:01:59.544649    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/111e1044-6683-459f-9e1f-3e0cb9b8fd4c-maas-api-tls\") on node \"ip-10-0-141-65.ec2.internal\" DevicePath \"\""
Jul 22 19:02:00.384030 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:00.383995    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"
Jul 22 19:02:00.384030 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:00.384017    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q" event={"ID":"111e1044-6683-459f-9e1f-3e0cb9b8fd4c","Type":"ContainerDied","Data":"6aef4afe3b0d610d1d801a22789e278b0db316db740c2bcddf140b9b8f42546b"}
Jul 22 19:02:00.384538 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:00.384084    2580 scope.go:117] "RemoveContainer" containerID="ba5a241944de83f357c13852a22e0d75c302739841266ae6cd4d177771312dd6"
Jul 22 19:02:00.407974 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:00.407952    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 19:02:00.411401 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:00.411373    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-63c82d-886559bb-d8z6q"]
Jul 22 19:02:01.886470 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:01.886435    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-wl4xl"]
Jul 22 19:02:02.320627 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:02.320592    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" path="/var/lib/kubelet/pods/111e1044-6683-459f-9e1f-3e0cb9b8fd4c/volumes"
Jul 22 19:02:03.827073 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:03.827018    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"]
Jul 22 19:02:03.829115 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:03.829096    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-63c82d-8gnlt"]
Jul 22 19:02:04.320765 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:04.320724    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76596da6-4932-4939-896b-0993d7b8cee1" path="/var/lib/kubelet/pods/76596da6-4932-4939-896b-0993d7b8cee1/volumes"
Jul 22 19:02:17.590251 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:17.590222    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5x9dd_ec746d59-da6e-44ab-bcb0-8c28d42de061/manager/0.log"
Jul 22 19:02:17.715160 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:17.715131    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gq8j_e2dbfcfa-8938-4fc2-9727-38c8fca47faa/manager/0.log"
Jul 22 19:02:17.968643 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:17.968617    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-m98r2_c8e501ed-a65b-48eb-834d-8d961a16c0a0/server/0.log"
Jul 22 19:02:18.093242 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:18.093219    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sfjkm_4b2169c5-3da0-43b2-9e01-3bb431aa940e/manager/0.log"
Jul 22 19:02:18.472967 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:18.472942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549865fcdf-pqbqc_5fd4abd6-c7f6-4cd1-b258-656ad70048fd/copy-manifests/0.log"
Jul 22 19:02:18.491317 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:18.491296    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549865fcdf-pqbqc_5fd4abd6-c7f6-4cd1-b258-656ad70048fd/manager/0.log"
Jul 22 19:02:18.985970 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:18.985944    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745750-hznl6_d10cd516-f596-436d-8f3a-dc8e98b0cade/cleanup/0.log"
Jul 22 19:02:19.103799 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:19.103777    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745765-6l79j_0fb905a0-bc69-490a-9445-5528e516b493/cleanup/0.log"
Jul 22 19:02:19.332695 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:19.332625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-6t9vv_f5560a20-0cc4-4e15-a1bb-7c2490209af0/postgres/0.log"
Jul 22 19:02:20.901188 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:20.901159    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nrs7h_ee03b86e-1f4e-49f1-8f3c-42168ebfaab3/manager/0.log"
Jul 22 19:02:21.029697 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:21.029673    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wjh69_919b3a0a-2915-4f52-8cf6-cae9adfa287a/manager/0.log"
Jul 22 19:02:21.147556 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:21.147534    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-85h4m_3f6a68df-04f0-4223-a91d-eca23e6ea3a0/kuadrant-console-plugin/0.log"
Jul 22 19:02:21.277731 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:21.277709    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jn9p8_fa89d916-c291-4fc3-87d7-605d18ae5cc9/registry-server/0.log"
Jul 22 19:02:21.539586 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:21.539515    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wl4xl_f63327da-6ba2-426f-bb92-03bdba85bc14/limitador/0.log"
Jul 22 19:02:21.674489 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:21.674472    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sxgdc_70e234f8-4190-4869-917a-8d185c61380a/manager/0.log"
Jul 22 19:02:22.334492 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:22.334466    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4c4dbc87-6x4kh_82946fd3-0ef7-4f51-8760-85524d2f27e5/kube-auth-proxy/0.log"
Jul 22 19:02:23.376036 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:23.376007    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-sbswv_f988ac90-ad1f-47fb-845e-9f79f68d937f/main/0.log"
Jul 22 19:02:23.498322 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:23.498291    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-ckvps_482f9edf-b110-4b38-b93e-152f865f0f92/main/0.log"
Jul 22 19:02:29.625360 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625328    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r6bg7/must-gather-zfg46"]
Jul 22 19:02:29.625734 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625654    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" containerName="maas-api"
Jul 22 19:02:29.625734 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625664    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" containerName="maas-api"
Jul 22 19:02:29.625734 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625690    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76596da6-4932-4939-896b-0993d7b8cee1" containerName="revoke-keys"
Jul 22 19:02:29.625734 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625695    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76596da6-4932-4939-896b-0993d7b8cee1" containerName="revoke-keys"
Jul 22 19:02:29.625899 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625743    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="111e1044-6683-459f-9e1f-3e0cb9b8fd4c" containerName="maas-api"
Jul 22 19:02:29.625899 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.625753    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76596da6-4932-4939-896b-0993d7b8cee1" containerName="revoke-keys"
Jul 22 19:02:29.628875 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.628858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.631802 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.631784    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r6bg7\"/\"openshift-service-ca.crt\""
Jul 22 19:02:29.631917 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.631826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r6bg7\"/\"default-dockercfg-m6ggt\""
Jul 22 19:02:29.631917 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.631838    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r6bg7\"/\"kube-root-ca.crt\""
Jul 22 19:02:29.641499 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.641478    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6bg7/must-gather-zfg46"]
Jul 22 19:02:29.679586 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.679556    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0e02e38-0827-4340-b780-da9c226943f8-must-gather-output\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.679693 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.679627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdfn\" (UniqueName: \"kubernetes.io/projected/c0e02e38-0827-4340-b780-da9c226943f8-kube-api-access-jhdfn\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.780225 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.780203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0e02e38-0827-4340-b780-da9c226943f8-must-gather-output\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.780337 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.780244    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdfn\" (UniqueName: \"kubernetes.io/projected/c0e02e38-0827-4340-b780-da9c226943f8-kube-api-access-jhdfn\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.780544 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.780525    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0e02e38-0827-4340-b780-da9c226943f8-must-gather-output\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.791078 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.791061    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdfn\" (UniqueName: \"kubernetes.io/projected/c0e02e38-0827-4340-b780-da9c226943f8-kube-api-access-jhdfn\") pod \"must-gather-zfg46\" (UID: \"c0e02e38-0827-4340-b780-da9c226943f8\") " pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:29.938534 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:29.938512    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6bg7/must-gather-zfg46"
Jul 22 19:02:30.084202 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:30.084174    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6bg7/must-gather-zfg46"]
Jul 22 19:02:30.086622 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:02:30.086594    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e02e38_0827_4340_b780_da9c226943f8.slice/crio-01a9e1ec7a6066a70c3bdd6c59df42ffa369c1cb03de2361d9cd2fe8dbb6efc1 WatchSource:0}: Error finding container 01a9e1ec7a6066a70c3bdd6c59df42ffa369c1cb03de2361d9cd2fe8dbb6efc1: Status 404 returned error can't find the container with id 01a9e1ec7a6066a70c3bdd6c59df42ffa369c1cb03de2361d9cd2fe8dbb6efc1
Jul 22 19:02:30.524258 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:30.524228    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6bg7/must-gather-zfg46" event={"ID":"c0e02e38-0827-4340-b780-da9c226943f8","Type":"ContainerStarted","Data":"01a9e1ec7a6066a70c3bdd6c59df42ffa369c1cb03de2361d9cd2fe8dbb6efc1"}
Jul 22 19:02:31.533640 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:31.533556    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6bg7/must-gather-zfg46" event={"ID":"c0e02e38-0827-4340-b780-da9c226943f8","Type":"ContainerStarted","Data":"893d250959cebc0c01dc877166387f9a192f929555f89190cc4f214dc6191350"}
Jul 22 19:02:32.543975 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:32.543930    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6bg7/must-gather-zfg46" event={"ID":"c0e02e38-0827-4340-b780-da9c226943f8","Type":"ContainerStarted","Data":"6c5a722bddf9aa60ced0a6cca5b9ea82d1f7ae5143f7a2e99981766798dc356b"}
Jul 22 19:02:32.560330 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:32.560281    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r6bg7/must-gather-zfg46" podStartSLOduration=2.230624243 podStartE2EDuration="3.56026454s" totalImagesPullingTime="1.329640297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 19:02:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:32.559397489 +0000 UTC m=+2906.775044893" watchObservedRunningTime="2026-07-22 19:02:32.56026454 +0000 UTC m=+2906.775911979"
Jul 22 19:02:33.020589 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:33.020561    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2zn89_60a152e6-60d1-4e08-86c5-8d03303c4cec/global-pull-secret-syncer/0.log"
Jul 22 19:02:33.278849 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:33.278760    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s2pqm_5a14c924-510b-4134-a5f1-751bc9baf02d/konnectivity-agent/0.log"
Jul 22 19:02:33.354608 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:33.354580    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-65.ec2.internal_b14abac64ed2c3ba96221ac6ba71509b/haproxy/0.log"
Jul 22 19:02:38.451987 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.451946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-nrs7h_ee03b86e-1f4e-49f1-8f3c-42168ebfaab3/manager/0.log"
Jul 22 19:02:38.476894 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.476847    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wjh69_919b3a0a-2915-4f52-8cf6-cae9adfa287a/manager/0.log"
Jul 22 19:02:38.507455 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.507422    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-85h4m_3f6a68df-04f0-4223-a91d-eca23e6ea3a0/kuadrant-console-plugin/0.log"
Jul 22 19:02:38.561606 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.561577    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-jn9p8_fa89d916-c291-4fc3-87d7-605d18ae5cc9/registry-server/0.log"
Jul 22 19:02:38.731503 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.731397    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-wl4xl_f63327da-6ba2-426f-bb92-03bdba85bc14/limitador/0.log"
Jul 22 19:02:38.821737 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:38.821704    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-sxgdc_70e234f8-4190-4869-917a-8d185c61380a/manager/0.log"
Jul 22 19:02:40.824222 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:40.824134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vwdbn_632f93a4-14f6-4feb-985a-d44f1510efc6/node-exporter/0.log"
Jul 22 19:02:40.846967 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:40.846941    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vwdbn_632f93a4-14f6-4feb-985a-d44f1510efc6/kube-rbac-proxy/0.log"
Jul 22 19:02:40.875614 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:40.875591    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vwdbn_632f93a4-14f6-4feb-985a-d44f1510efc6/init-textfile/0.log"
Jul 22 19:02:41.234654 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.234625    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-dj4vr_86f2a168-648b-40f9-a1e3-ea8e337a3dc8/prometheus-operator-admission-webhook/0.log"
Jul 22 19:02:41.338063 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.338033    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/thanos-query/0.log"
Jul 22 19:02:41.358378 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.358354    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/kube-rbac-proxy-web/0.log"
Jul 22 19:02:41.379435 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.379405    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/kube-rbac-proxy/0.log"
Jul 22 19:02:41.400688 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.400659    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/prom-label-proxy/0.log"
Jul 22 19:02:41.430626 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.430560    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/kube-rbac-proxy-rules/0.log"
Jul 22 19:02:41.451768 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:41.451709    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bfff888c-k4cbx_43165952-70c1-46cd-8180-22828d6af0c8/kube-rbac-proxy-metrics/0.log"
Jul 22 19:02:43.146264 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.146228    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"]
Jul 22 19:02:43.152499 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.152468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.158255 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.158221    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"]
Jul 22 19:02:43.306769 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.306722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-proc\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.307003 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.306777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-podres\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.307003 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.306890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-lib-modules\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.307003 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.306996    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnk8s\" (UniqueName: \"kubernetes.io/projected/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-kube-api-access-rnk8s\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.307190 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.307028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-sys\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408306 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408210    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-proc\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408306 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-podres\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408354    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-proc\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-lib-modules\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408399    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-podres\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408458    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-lib-modules\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408494    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnk8s\" (UniqueName: \"kubernetes.io/projected/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-kube-api-access-rnk8s\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408542 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-sys\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.408893 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.408665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-sys\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.417537 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.417511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnk8s\" (UniqueName: \"kubernetes.io/projected/b4bb9fd9-c2ae-476a-a607-c6c76214e5f2-kube-api-access-rnk8s\") pod \"perf-node-gather-daemonset-qtcvl\" (UID: \"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2\") " pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.469017 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.468981    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:43.657364 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.657331    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"]
Jul 22 19:02:43.661032 ip-10-0-141-65 kubenswrapper[2580]: W0722 19:02:43.661002    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4bb9fd9_c2ae_476a_a607_c6c76214e5f2.slice/crio-b2b8ed5d9fb514bb854851cc5bf331d9a6efd88c0dbc1153581ab949ab314300 WatchSource:0}: Error finding container b2b8ed5d9fb514bb854851cc5bf331d9a6efd88c0dbc1153581ab949ab314300: Status 404 returned error can't find the container with id b2b8ed5d9fb514bb854851cc5bf331d9a6efd88c0dbc1153581ab949ab314300
Jul 22 19:02:43.832448 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:43.832420    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f79954d97-dnh22_51a7d91a-dcd8-4c1c-859a-c022a37c3b2e/console/0.log"
Jul 22 19:02:44.627044 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:44.627010    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl" event={"ID":"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2","Type":"ContainerStarted","Data":"88d2106805ec311297500625a764d421e2e57e57ca742b2c2db100c24286b602"}
Jul 22 19:02:44.627463 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:44.627072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl" event={"ID":"b4bb9fd9-c2ae-476a-a607-c6c76214e5f2","Type":"ContainerStarted","Data":"b2b8ed5d9fb514bb854851cc5bf331d9a6efd88c0dbc1153581ab949ab314300"}
Jul 22 19:02:44.627463 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:44.627165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:44.657712 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:44.657661    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl" podStartSLOduration=1.657645641 podStartE2EDuration="1.657645641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 19:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 19:02:44.65509867 +0000 UTC m=+2918.870746075" watchObservedRunningTime="2026-07-22 19:02:44.657645641 +0000 UTC m=+2918.873293044"
Jul 22 19:02:45.483839 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:45.483794    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bthdp_f06502df-814d-476e-9f23-9eaae5e32a1d/dns/0.log"
Jul 22 19:02:45.507062 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:45.507027    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bthdp_f06502df-814d-476e-9f23-9eaae5e32a1d/kube-rbac-proxy/0.log"
Jul 22 19:02:45.697538 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:45.697506    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jxvp4_16f3208c-6065-4712-9296-87de0e812da9/dns-node-resolver/0.log"
Jul 22 19:02:46.282768 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:46.282739    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mmgns_6ae127f4-a41c-4faa-8156-7ec3d2c46382/node-ca/0.log"
Jul 22 19:02:47.406070 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:47.406039    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4c4dbc87-6x4kh_82946fd3-0ef7-4f51-8760-85524d2f27e5/kube-auth-proxy/0.log"
Jul 22 19:02:48.294558 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:48.294526    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgxlm_107e48ae-5663-4d7d-9d6d-a26d5270144c/serve-healthcheck-canary/0.log"
Jul 22 19:02:49.060518 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:49.060492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svp7h_26ddf080-af95-493c-96b0-802746aa037e/kube-rbac-proxy/0.log"
Jul 22 19:02:49.099680 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:49.099646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svp7h_26ddf080-af95-493c-96b0-802746aa037e/exporter/0.log"
Jul 22 19:02:49.152387 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:49.152357    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-svp7h_26ddf080-af95-493c-96b0-802746aa037e/extractor/0.log"
Jul 22 19:02:50.644008 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:50.643978    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r6bg7/perf-node-gather-daemonset-qtcvl"
Jul 22 19:02:51.686133 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:51.686088    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5x9dd_ec746d59-da6e-44ab-bcb0-8c28d42de061/manager/0.log"
Jul 22 19:02:51.960667 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:51.960585    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7gq8j_e2dbfcfa-8938-4fc2-9727-38c8fca47faa/manager/0.log"
Jul 22 19:02:52.104756 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:52.104723    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-m98r2_c8e501ed-a65b-48eb-834d-8d961a16c0a0/server/0.log"
Jul 22 19:02:52.197070 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:52.197036    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-sfjkm_4b2169c5-3da0-43b2-9e01-3bb431aa940e/manager/0.log"
Jul 22 19:02:52.419116 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:52.418968    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549865fcdf-pqbqc_5fd4abd6-c7f6-4cd1-b258-656ad70048fd/manager/0.log"
Jul 22 19:02:52.438327 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:52.438297    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-549865fcdf-pqbqc_5fd4abd6-c7f6-4cd1-b258-656ad70048fd/copy-manifests/0.log"
Jul 22 19:02:53.731930 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:53.731900    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4c4dbc87-6x4kh_82946fd3-0ef7-4f51-8760-85524d2f27e5/kube-auth-proxy/0.log"
Jul 22 19:02:55.309373 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:02:55.309295    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f4c4dbc87-6x4kh_82946fd3-0ef7-4f51-8760-85524d2f27e5/kube-auth-proxy/0.log"
Jul 22 19:03:00.117112 ip-10-0-141-65 kubenswrapper[2580]: I0722 19:03:00.117074    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9tl6j_f5c99fd1-d6ff-4257-a696-e3905160793d/kube-multus/0.log"