Jul 27 15:48:22.972765 ip-10-0-135-171 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 15:48:23.373151 ip-10-0-135-171 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 15:48:23.373151 ip-10-0-135-171 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 15:48:23.373151 ip-10-0-135-171 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 15:48:23.373151 ip-10-0-135-171 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 15:48:23.373151 ip-10-0-135-171 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 15:48:23.374743 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.374658    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 15:48:23.381027 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381012    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.381027 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381026    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381031    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381034    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381037    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381040    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381043    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381046    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381049    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381057    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381060    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381063    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381066    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381068    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381071    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381073    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381076    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381078    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381081    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381083    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381086    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.381095 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381090    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381094    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381097    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381099    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381102    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381105    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381108    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381111    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381114    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381117    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381119    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381122    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381125    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381127    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381130    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381132    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381136    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381139    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381143    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.381560 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381146    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381149    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381152    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381155    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381158    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381162    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381165    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381168    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381170    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381173    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381175    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381178    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381180    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381182    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381185    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381188    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381191    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381193    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381196    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.382076 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381199    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381201    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381204    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381207    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381209    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381212    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381214    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381217    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381219    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381222    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381225    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381227    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381230    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381233    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381236    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381239    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381242    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381244    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381247    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.382529 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381249    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381252    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381254    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381257    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381259    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381262    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381264    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381267    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381269    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381274    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381277    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381279    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381282    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381284    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381287    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381290    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381292    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381295    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381297    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381300    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.383007 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381302    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381964    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381970    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381973    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381976    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381979    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381982    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.381984    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382003    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382008    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382013    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382017    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382021    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382024    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382027    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382029    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382032    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382035    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382037    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.383506 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382040    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382042    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382045    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382048    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382051    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382053    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382056    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382058    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382061    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382064    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382068    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382072    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382074    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382077    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382082    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382085    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382088    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382091    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382094    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.384043 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382097    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382102    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382105    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382108    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382111    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382118    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382121    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382124    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382127    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382131    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382134    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382136    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382140    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382143    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382146    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382149    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382151    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382154    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382157    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382160    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.384499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382162    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382165    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382167    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382170    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382172    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382175    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382178    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382180    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382183    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382185    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382188    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382190    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382194    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382196    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382199    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382201    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382204    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382207    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382213    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382216    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.385052 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382218    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382221    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382223    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382226    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382228    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382231    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382234    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382236    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382240    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382242    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382245    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382247    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382250    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382252    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382255    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382257    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382260    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382262    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382265    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382268    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.385551 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382270    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382272    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382672    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382677    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382681    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382683    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382686    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382688    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382691    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382694    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382704    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382707    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382709    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382712    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382714    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382717    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382720    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382722    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382725    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.386057 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382728    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382731    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382733    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382735    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382738    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382740    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382743    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382746    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382748    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382750    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382753    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382756    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382758    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382761    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382763    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382766    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382770    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382774    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382777    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382780    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.386518 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382783    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382785    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382788    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382790    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382798    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382801    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382803    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382806    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382808    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382811    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382813    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382816    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382819    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382821    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382824    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382826    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382829    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382832    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382834    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382837    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.387024 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382840    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382842    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382845    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382847    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382850    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382853    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382855    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382858    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382862    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382865    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382868    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382870    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382874    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382877    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382880    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382883    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382885    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382888    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382891    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382894    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.387526 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382896    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382899    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382901    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382904    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382907    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382909    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382912    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382914    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382917    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382923    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382925    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382928    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382930    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382933    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382936    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382939    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382942    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382944    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382947    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.388047 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382950    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382952    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.382955    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383031    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383039    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383057    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383063    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383070    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383073    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383077    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383081    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383085    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383089    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383092    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383095    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383098    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383101    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383104    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383107    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383110    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383113    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383116    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383120    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 15:48:23.388499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383123    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383126    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383129    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383133    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383136    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383139    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383142    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383145    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383148    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383151    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383154    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383157    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383160    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383163    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383166    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383169    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383172    2576 flags.go:64] FLAG: --enable-server="true"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383175    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383180    2576 flags.go:64] FLAG: --event-burst="100"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383183    2576 flags.go:64] FLAG: --event-qps="50"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383186    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383206    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383210    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383221    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 15:48:23.389062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383224    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383229    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383232    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383235    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383238    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383241    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383243    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383247    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383250    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383253    2576 flags.go:64] FLAG: --feature-gates=""
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383257    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383260    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383263    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383266    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383269    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383272    2576 flags.go:64] FLAG: --help="false"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383275    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-171.ec2.internal"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383278    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383280    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383283    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383286    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383289    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383292    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 15:48:23.389621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383295    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383298    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383300    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383303    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383307    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383309    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383312    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383315    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383319    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383321    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383326    2576 flags.go:64] FLAG: --lock-file=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383328    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383331    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383334    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383339    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383342    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383345    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383348    2576 flags.go:64] FLAG: --logging-format="text"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383352    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383355    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383358    2576 flags.go:64] FLAG: --manifest-url=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383360    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383364    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383367    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383371    2576 flags.go:64] FLAG: --max-pods="110"
Jul 27 15:48:23.390272 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383374    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383377    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383380    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383382    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383385    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383388    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383391    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383397    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383400    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383403    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383406    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383410    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383415    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383418    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383421    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383425    2576 flags.go:64] FLAG: --port="10250"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383428    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383430    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0db3b8b2d256b1c31"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383435    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383437    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383440    2576 flags.go:64] FLAG: --register-node="true"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383443    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383446    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383449    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 27 15:48:23.390885 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383452    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383455    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383458    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383461    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383464    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383467    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383470    2576 flags.go:64] FLAG: --runonce="false"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383472    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383475    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383478    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383481    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383484    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383487    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383490    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383492    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383495    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383498    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383501    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383505    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383508    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383511    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383514    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383519    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383522    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383525    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 15:48:23.391478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383529    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383533    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383536    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383538    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383541    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383544    2576 flags.go:64] FLAG: --v="2"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383548    2576 flags.go:64] FLAG: --version="false"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383552    2576 flags.go:64] FLAG: --vmodule=""
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383555    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.383558    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383702    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383707    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383710    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383713    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383716    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383718    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383721    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383723    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383726    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383730    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383733    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383736    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.392081 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383739    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383742    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383745    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383749    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383752    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383754    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383757    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383760    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383762    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383765    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383768    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383772    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383775    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383778    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383781    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383783    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383786    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383788    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383791    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.392617 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383793    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383796    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383799    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383801    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383804    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383806    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383809    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383811    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383814    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383817    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383819    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383822    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383824    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383828    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383831    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383835    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383840    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383843    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383845    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.393078 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383848    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383850    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383853    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383855    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383858    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383864    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383867    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383869    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383872    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383875    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383877    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383880    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383890    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383893    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383896    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383899    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383901    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383904    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383906    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383909    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.393524 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383912    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383914    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383917    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383920    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383923    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383925    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383928    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383930    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383933    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383936    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383939    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383942    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383944    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383947    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383949    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383952    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383955    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383958    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383962    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383965    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.394066 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383967    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383970    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383972    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383975    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383978    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383980    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383983    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.383997    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.384000    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.384557    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.391149    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.391162    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.391892    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.391901    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 15:48:23.394545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391956    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391961    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391964    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391967    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391970    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391973    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391975    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391978    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391981    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.391983    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392000    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392003    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392007    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392011    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392014    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392016    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392020    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392023    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392025    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392028    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.394893 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392031    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392033    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392036    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392039    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392041    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392044    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392047    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392049    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392052    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392055    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392058    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392062    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392066    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392070    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392073    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392076    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392079    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392081    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392084    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.395407 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392087    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392090    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392093    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392095    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392098    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392102    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392106    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392109    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392112    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392114    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392119    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392122    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392125    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392128    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392130    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392132    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392135    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392137    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392140    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392143    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.395866 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392145    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392148    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392151    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392154    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392157    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392159    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392162    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392164    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392167    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392169    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392172    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392174    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392177    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392179    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392182    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392185    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392187    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392189    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392192    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392195    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.396368 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392197    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392201    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392204    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392208    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392211    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392214    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392216    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392219    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392221    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392224    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392226    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392229    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392231    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392234    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392237    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392240    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392243    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392245    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392247    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.396849 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392250    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.392255    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392363    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392367    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392370    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392373    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392376    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392378    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392381    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392383    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392386    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392388    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392391    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392393    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 15:48:23.397338 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392396    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392398    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392401    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392404    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392407    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392409    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392412    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392414    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392417    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392419    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392422    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392424    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392427    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392429    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392432    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392434    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392437    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392439    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392442    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392444    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 15:48:23.397672 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392447    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392449    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392452    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392454    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392457    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392459    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392461    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392464    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392466    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392469    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392472    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392474    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392476    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392479    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392481    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392484    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392487    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392489    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392491    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392494    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 15:48:23.398270 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392496    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392499    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392501    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392504    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392506    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392509    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392511    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392514    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392516    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392519    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392521    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392524    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392526    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392529    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392531    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392534    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392536    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392539    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392541    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392544    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 15:48:23.398753 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392546    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392549    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392551    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392553    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392556    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392560    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392563    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392566    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392569    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392574    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392577    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392580    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392583    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392586    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392588    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392591    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392594    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392596    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392599    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392601    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 15:48:23.399259 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392604    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392606    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392609    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392612    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392616    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392618    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:23.392621    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.392626    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.392721    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.394736    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.396097    2576 server.go:1019] "Starting client certificate rotation"
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.396195    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.396231    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 15:48:23.399733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.398248    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 15:48:23.402006 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.401967    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 15:48:23.415853 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.415834    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 27 15:48:23.417853 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.417839    2576 log.go:25] "Validated CRI v1 image API"
Jul 27 15:48:23.420127 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.420112    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 15:48:23.422326 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.422301    2576 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a4c91077-36a1-45a0-bffe-5e4b523471b1:/dev/nvme0n1p4 bd51b130-cd35-471c-bfb3-a9ff23746f16:/dev/nvme0n1p3]
Jul 27 15:48:23.422388 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.422326    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 15:48:23.425657 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.425640    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 15:48:23.427223 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.427105    2576 manager.go:216] Machine: {Timestamp:2026-07-27 15:48:23.425976731 +0000 UTC m=+0.342146012 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098673 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ea1bf37145b6c2c47ac29515d4625 SystemUUID:ec2ea1bf-3714-5b6c-2c47-ac29515d4625 BootID:2c1f2332-25e2-41b6-ab57-e43558faf141 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:50:aa:77:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:50:aa:77:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:53:39:8c:19:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 15:48:23.427223 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.427218    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 15:48:23.427334 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.427290    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 15:48:23.430338 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.430317    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 15:48:23.430465 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.430342    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-171.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 15:48:23.431077 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.431067    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 15:48:23.431114 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.431079    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 15:48:23.431114 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.431090    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 15:48:23.431114 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.431103    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 15:48:23.432010 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.431985    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 15:48:23.434133 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434122    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 15:48:23.434238 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434228    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 15:48:23.434274 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434256    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 15:48:23.434274 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434266    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 15:48:23.434326 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434293    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 15:48:23.434326 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434303    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 27 15:48:23.434326 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434317    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 15:48:23.434622 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434605    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 15:48:23.434665 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.434626    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 15:48:23.436808 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.436793    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 15:48:23.438672 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.438656    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 15:48:23.438736 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.438687    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 15:48:23.440390 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440378    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440398    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440405    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440410    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440419    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440425    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440433    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 15:48:23.440437 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440438    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440444    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440450    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440465    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440470    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440496    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 15:48:23.440618 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.440501    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 15:48:23.443727 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.443710    2576 server.go:1268] "Started kubelet"
Jul 27 15:48:23.443824 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.443796    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 15:48:23.446870 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.446847    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 15:48:23.447134 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.443826    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 15:48:23.447276 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.447247    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 15:48:23.447339 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.447289    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-171.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 15:48:23.447455 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.447435    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 15:48:23.447457 ip-10-0-135-171 systemd[1]: Started Kubernetes Kubelet.
Jul 27 15:48:23.449909 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.449480    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 15:48:23.450967 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.450949    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 15:48:23.451714 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.451697    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56jmc"
Jul 27 15:48:23.452374 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.452356    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 15:48:23.454383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.453415    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-171.ec2.internal.18c630344a672496  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-171.ec2.internal,UID:ip-10-0-135-171.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-171.ec2.internal,},FirstTimestamp:2026-07-27 15:48:23.44368655 +0000 UTC m=+0.359855831,LastTimestamp:2026-07-27 15:48:23.44368655 +0000 UTC m=+0.359855831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-171.ec2.internal,}"
Jul 27 15:48:23.457907 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.457889    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 15:48:23.457907 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.457897    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 15:48:23.458073 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.457983    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:23.458660 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458644    2576 factory.go:55] Registering systemd factory
Jul 27 15:48:23.458660 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458663    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 27 15:48:23.458794 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458710    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 15:48:23.458794 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458723    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 15:48:23.458794 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458774    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 15:48:23.458914 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458859    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 15:48:23.458914 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458887    2576 factory.go:153] Registering CRI-O factory
Jul 27 15:48:23.458914 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458901    2576 factory.go:223] Registration of the crio container factory successfully
Jul 27 15:48:23.459056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458888    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 15:48:23.459056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.458962    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 15:48:23.459056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.459030    2576 factory.go:103] Registering Raw factory
Jul 27 15:48:23.459056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.459044    2576 manager.go:1196] Started watching for new ooms in manager
Jul 27 15:48:23.459378 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.459366    2576 manager.go:349] Starting recovery of all containers
Jul 27 15:48:23.459503 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.459487    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-56jmc"
Jul 27 15:48:23.461354 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.461326    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-171.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 15:48:23.461448 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:23.461405    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 15:48:23.469257 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.469240    2576 manager.go:354] Recovery completed
Jul 27 15:48:23.473354 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.473305    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:23.475304 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475289    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:23.475378 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475317    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:23.475378 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475330    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:23.475849 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475835    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 15:48:23.475849 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475849    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:23.475959 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475849    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 15:48:23.475959 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.475900    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 15:48:23.477098 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477081    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:23.477155 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477115    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:23.477155 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477130    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:23.477616 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477604    2576 policy_none.go:49] "None policy: Start"
Jul 27 15:48:23.477648 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477621    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 15:48:23.477648 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.477631    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 15:48:23.478653 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:23.478641    2576 policy_none.go:47] "Start"
Jul 27 15:48:24.525728 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.525621    2576 manager.go:337] "Starting Device Plugin manager"
Jul 27 15:48:24.525810 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.525745    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 15:48:24.525810 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.525758    2576 server.go:87] "Starting device plugin registration server"
Jul 27 15:48:24.525987 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.525974    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 15:48:24.526055 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.525988    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 15:48:24.526119 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.526083    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 15:48:24.526208 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.526182    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 15:48:24.526208 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.526193    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 15:48:24.526592 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.526567    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 15:48:24.526672 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.526623    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:24.626350 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.626301    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:24.627416 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.627145    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:24.627416 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.627170    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:24.627416 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.627179    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:24.627416 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.627205    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.634846 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.634832    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.634896 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.634853    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-171.ec2.internal\": node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:24.644735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.644709    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 15:48:24.645913 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.645893    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 15:48:24.645978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.645941    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 15:48:24.645978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.645965    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 15:48:24.646063 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.646011    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 15:48:24.647779 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.647765    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:24.651838 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.651823    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:24.746385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.746364    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"]
Jul 27 15:48:24.746460 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.746421    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:24.747220 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.747203    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:24.747288 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.747230    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:24.747288 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.747243    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:24.748536 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.748524    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:24.748644 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.748627    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.748644 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.748652    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:24.749444 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749429    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:24.749534 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749450    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:24.749534 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749460    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:24.749534 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749468    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:24.749534 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749484    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:24.749534 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.749493    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:24.750718 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.750702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.750779 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.750735    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 15:48:24.751914 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.751899    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:24.752011 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.751998    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 15:48:24.752070 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.752027    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 15:48:24.752070 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.752042    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeHasSufficientPID"
Jul 27 15:48:24.763097 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.763081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe185acdf451816f7e23b5f888d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-171.ec2.internal\" (UID: \"9a03fbe185acdf451816f7e23b5f888d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.763200 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.763114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.763200 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.763134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.777586 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.777568    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-171.ec2.internal\" not found" node="ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.781900 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.781887    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-171.ec2.internal\" not found" node="ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.852076 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.852058    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:24.863641 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.863730 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.863730 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe185acdf451816f7e23b5f888d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-171.ec2.internal\" (UID: \"9a03fbe185acdf451816f7e23b5f888d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.863730 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe185acdf451816f7e23b5f888d-config\") pod \"kube-apiserver-proxy-ip-10-0-135-171.ec2.internal\" (UID: \"9a03fbe185acdf451816f7e23b5f888d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.863730 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863722    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.863872 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:24.863739    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6570cf7faba16b53fa79128551a488d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal\" (UID: \"d6570cf7faba16b53fa79128551a488d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:24.953081 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:24.953030    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:25.053552 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.053508    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:25.080044 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.080018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:25.084819 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.084804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:25.154259 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.154229    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:25.254856 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.254783    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-171.ec2.internal\" not found"
Jul 27 15:48:25.306370 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.306349    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:25.360807 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.360782    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:25.368340 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.368317    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 15:48:25.370259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.370244    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal"
Jul 27 15:48:25.380507 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.380494    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 15:48:25.397649 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.397635    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 15:48:25.397758 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.397741    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 15:48:25.397797 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.397743    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 15:48:25.436941 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.436917    2576 apiserver.go:52] "Watching apiserver"
Jul 27 15:48:25.453130 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.453095    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 15:48:25.453446 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.453427    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-2lzpz","openshift-network-diagnostics/network-check-target-49g5r","openshift-network-operator/iptables-alerter-76t4v","openshift-ovn-kubernetes/ovnkube-node-qh8s2","kube-system/global-pull-secret-syncer-qd4fh","kube-system/konnectivity-agent-dp29f","kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz","openshift-cluster-node-tuning-operator/tuned-cdj97","openshift-dns/node-resolver-kdn7c","openshift-image-registry/node-ca-sst8w","openshift-multus/multus-additional-cni-plugins-mmk5r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal","openshift-multus/multus-kjkq8"]
Jul 27 15:48:25.454980 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.454963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.456062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.456045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.457867 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.457652    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.457867 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.457722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.457867 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.457732    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fz2mj\""
Jul 27 15:48:25.457867 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.457734    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.459503 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.459484    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.459913 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.459896    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.460001 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.459983    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5knh4\""
Jul 27 15:48:25.460001 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.459992    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.460228 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.460209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 15:48:25.460352 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.459897    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 15:48:25.460795 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.460781    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 15:48:25.461298 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.461239    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r8k6z\""
Jul 27 15:48:25.461405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.461390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.461470 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.461413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.461518 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.461493    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.461518 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.461482    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:25.462261 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.462241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 15:48:25.462432 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.462418    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 15:48:25.462737 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.462721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.463158 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.463136    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.463266 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.463250    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 15:48:25.463815 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.463801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 15:48:25.463858 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.463806    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8lbtx\""
Jul 27 15:48:25.463858 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.463827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.464070 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.464055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.464921 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.464906    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v822k\""
Jul 27 15:48:25.464978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.464927    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 15:48:25.464978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.464929    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 15:48:25.465175 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.465162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.466307 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466276    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 15:43:24 +0000 UTC" deadline="2028-02-22 19:36:03.009323852 +0000 UTC"
Jul 27 15:48:25.466307 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466306    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13803h47m37.543020023s"
Jul 27 15:48:25.466493 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466478    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:25.466567 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.466552    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:25.466727 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.466814 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466737    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t4shq\""
Jul 27 15:48:25.466814 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.466775    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.467098 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-config\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467186 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1da3b24-14ce-4aa1-96ff-b045734336da-tmp-dir\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.467186 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/810304a7-7ea9-4638-a126-10f6765aabca-host-slash\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.467186 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-env-overrides\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467186 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-script-lib\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.467366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73411ea9-8cbf-44ed-b28d-b1dba00cd762-serviceca\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.467366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-log-socket\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-systemd-units\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-kubelet-config\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1da3b24-14ce-4aa1-96ff-b045734336da-hosts-file\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73411ea9-8cbf-44ed-b28d-b1dba00cd762-host\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6l8\" (UniqueName: \"kubernetes.io/projected/73411ea9-8cbf-44ed-b28d-b1dba00cd762-kube-api-access-6v6l8\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467496    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/810304a7-7ea9-4638-a126-10f6765aabca-iptables-alerter-script\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.467543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-kubelet\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-slash\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-netns\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-netd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467733    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbr6\" (UniqueName: \"kubernetes.io/projected/810304a7-7ea9-4638-a126-10f6765aabca-kube-api-access-qwbr6\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-var-lib-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467818    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-etc-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467747    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lsq2h\""
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-ovn\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-dbus\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.467898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224xb\" (UniqueName: \"kubernetes.io/projected/955df176-6f69-4291-9ffa-25dd76c3f1ab-kube-api-access-224xb\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd0534f6-4899-4330-a551-9c446059d5e8-agent-certs\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-node-log\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467697    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.467962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7stxk\" (UniqueName: \"kubernetes.io/projected/a1da3b24-14ce-4aa1-96ff-b045734336da-kube-api-access-7stxk\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-systemd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468040    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd0534f6-4899-4330-a551-9c446059d5e8-konnectivity-ca\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468071    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-bin\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.468191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.468441 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.468251    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:25.469424 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.469405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.470874 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.470857    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.472020 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.471525    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 15:48:25.472020 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.471604    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 15:48:25.472020 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.471647    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qvrlv\""
Jul 27 15:48:25.472020 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.471657    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 15:48:25.472470 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.472456    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 15:48:25.473097 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.473083    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7fhkl\""
Jul 27 15:48:25.473178 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.473116    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 15:48:25.473178 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.473097    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 15:48:25.498320 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.498302    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4n64"
Jul 27 15:48:25.509062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.509018    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h4n64"
Jul 27 15:48:25.561807 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.561786    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 15:48:25.566446 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.566411    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:25.568299 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/810304a7-7ea9-4638-a126-10f6765aabca-iptables-alerter-script\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.568385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-kubelet\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-slash\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-netns\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-dbus\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.568385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-socket-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd0534f6-4899-4330-a551-9c446059d5e8-agent-certs\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568396    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-kubelet\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-netns\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-slash\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-netd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbr6\" (UniqueName: \"kubernetes.io/projected/810304a7-7ea9-4638-a126-10f6765aabca-kube-api-access-qwbr6\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-dbus\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-netd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-etc-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-ovn\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.568631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-multus-certs\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqq5b\" (UniqueName: \"kubernetes.io/projected/5c391c2f-95c7-491f-800c-2071ad9d4887-kube-api-access-xqq5b\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-etc-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224xb\" (UniqueName: \"kubernetes.io/projected/955df176-6f69-4291-9ffa-25dd76c3f1ab-kube-api-access-224xb\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-ovn\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568691    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-registration-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-kubernetes\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-var-lib-kubelet\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-kubelet\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568756    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568805    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-multus-daemon-config\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-systemd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568901    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/810304a7-7ea9-4638-a126-10f6765aabca-iptables-alerter-script\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-systemd\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.568986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd0534f6-4899-4330-a551-9c446059d5e8-konnectivity-ca\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-bin\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-config\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1da3b24-14ce-4aa1-96ff-b045734336da-tmp-dir\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-cni-bin\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-modprobe-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-conf-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-system-cni-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.569227    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.569319    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:26.069277197 +0000 UTC m=+1.983333726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73411ea9-8cbf-44ed-b28d-b1dba00cd762-serviceca\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569490    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dd0534f6-4899-4330-a551-9c446059d5e8-konnectivity-ca\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-etc-kubernetes\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-run-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.569966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-device-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kps\" (UniqueName: \"kubernetes.io/projected/08500cb6-32e2-49b8-bdaf-cad7b1599248-kube-api-access-d5kps\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73411ea9-8cbf-44ed-b28d-b1dba00cd762-serviceca\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-systemd-units\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-run\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-systemd-units\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-host\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1da3b24-14ce-4aa1-96ff-b045734336da-tmp-dir\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-tuned\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-os-release\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1da3b24-14ce-4aa1-96ff-b045734336da-hosts-file\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73411ea9-8cbf-44ed-b28d-b1dba00cd762-host\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1da3b24-14ce-4aa1-96ff-b045734336da-hosts-file\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6l8\" (UniqueName: \"kubernetes.io/projected/73411ea9-8cbf-44ed-b28d-b1dba00cd762-kube-api-access-6v6l8\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.570735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73411ea9-8cbf-44ed-b28d-b1dba00cd762-host\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-multus\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-cnibin\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.569977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-systemd\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-var-lib-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-conf\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-sys\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-host-run-ovn-kubernetes\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-cni-binary-copy\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-hostroot\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-var-lib-openvswitch\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-node-log\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-config\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570259    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-node-log\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571362 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysconfig\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-lib-modules\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzw4p\" (UniqueName: \"kubernetes.io/projected/91f2519d-05ff-45e7-a6ea-9b929600c64e-kube-api-access-gzw4p\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-system-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-k8s-cni-cncf-io\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7stxk\" (UniqueName: \"kubernetes.io/projected/a1da3b24-14ce-4aa1-96ff-b045734336da-kube-api-access-7stxk\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-tmp\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx4nc\" (UniqueName: \"kubernetes.io/projected/d035bba7-a661-4e73-ad2a-83643f775471-kube-api-access-xx4nc\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-sys-fs\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/810304a7-7ea9-4638-a126-10f6765aabca-host-slash\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-env-overrides\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-script-lib\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-bin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/810304a7-7ea9-4638-a126-10f6765aabca-host-slash\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-log-socket\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-cnibin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-netns\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.571817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570826    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/955df176-6f69-4291-9ffa-25dd76c3f1ab-log-socket\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpv6v\" (UniqueName: \"kubernetes.io/projected/c549b894-3421-4622-8f97-d756354ab0f8-kube-api-access-lpv6v\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-kubelet-config\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-socket-dir-parent\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570982    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-kubelet-config\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.570982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.571027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-env-overrides\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.571017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-os-release\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.571061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.571302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovnkube-script-lib\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.572022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/955df176-6f69-4291-9ffa-25dd76c3f1ab-ovn-node-metrics-cert\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.572293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.572149    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dd0534f6-4899-4330-a551-9c446059d5e8-agent-certs\") pod \"konnectivity-agent-dp29f\" (UID: \"dd0534f6-4899-4330-a551-9c446059d5e8\") " pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.579616 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.579596    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-224xb\" (UniqueName: \"kubernetes.io/projected/955df176-6f69-4291-9ffa-25dd76c3f1ab-kube-api-access-224xb\") pod \"ovnkube-node-qh8s2\" (UID: \"955df176-6f69-4291-9ffa-25dd76c3f1ab\") " pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.580278 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.580257    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbr6\" (UniqueName: \"kubernetes.io/projected/810304a7-7ea9-4638-a126-10f6765aabca-kube-api-access-qwbr6\") pod \"iptables-alerter-76t4v\" (UID: \"810304a7-7ea9-4638-a126-10f6765aabca\") " pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.580487 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.580465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6l8\" (UniqueName: \"kubernetes.io/projected/73411ea9-8cbf-44ed-b28d-b1dba00cd762-kube-api-access-6v6l8\") pod \"node-ca-sst8w\" (UID: \"73411ea9-8cbf-44ed-b28d-b1dba00cd762\") " pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.580700 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.580685    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7stxk\" (UniqueName: \"kubernetes.io/projected/a1da3b24-14ce-4aa1-96ff-b045734336da-kube-api-access-7stxk\") pod \"node-resolver-kdn7c\" (UID: \"a1da3b24-14ce-4aa1-96ff-b045734336da\") " pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.600141 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.600113    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6570cf7faba16b53fa79128551a488d.slice/crio-6d9a805db7d5d0b7258e67a03414d15d392355dd69190622e4bc0c6bce30dddd WatchSource:0}: Error finding container 6d9a805db7d5d0b7258e67a03414d15d392355dd69190622e4bc0c6bce30dddd: Status 404 returned error can't find the container with id 6d9a805db7d5d0b7258e67a03414d15d392355dd69190622e4bc0c6bce30dddd
Jul 27 15:48:25.604091 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.604078    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 15:48:25.652013 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.651974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal" event={"ID":"d6570cf7faba16b53fa79128551a488d","Type":"ContainerStarted","Data":"6d9a805db7d5d0b7258e67a03414d15d392355dd69190622e4bc0c6bce30dddd"}
Jul 27 15:48:25.671459 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-conf\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671544 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-sys\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671544 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-cni-binary-copy\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671544 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-hostroot\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-sys\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-conf\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-hostroot\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671650    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysconfig\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-lib-modules\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzw4p\" (UniqueName: \"kubernetes.io/projected/91f2519d-05ff-45e7-a6ea-9b929600c64e-kube-api-access-gzw4p\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671733    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysconfig\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-system-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-lib-modules\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671828    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-k8s-cni-cncf-io\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-tmp\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671859    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-system-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx4nc\" (UniqueName: \"kubernetes.io/projected/d035bba7-a661-4e73-ad2a-83643f775471-kube-api-access-xx4nc\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-k8s-cni-cncf-io\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-sys-fs\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-bin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-cnibin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.671977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-netns\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpv6v\" (UniqueName: \"kubernetes.io/projected/c549b894-3421-4622-8f97-d756354ab0f8-kube-api-access-lpv6v\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-socket-dir-parent\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672032    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-cnibin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-bin\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.671995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-sys-fs\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672090    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-os-release\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672136    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-cni-binary-copy\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-socket-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672208    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-os-release\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-multus-certs\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672222    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-cni-binary-copy\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.672595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-netns\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-socket-dir-parent\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-run-multus-certs\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-etc-selinux\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqq5b\" (UniqueName: \"kubernetes.io/projected/5c391c2f-95c7-491f-800c-2071ad9d4887-kube-api-access-xqq5b\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-registration-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-cni-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-kubernetes\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-socket-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-var-lib-kubelet\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-kubelet\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672391    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-registration-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-multus-daemon-config\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-sysctl-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-kubernetes\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-var-lib-kubelet\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-kubelet\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-modprobe-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.672596    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-conf-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.672642    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:26.172626012 +0000 UTC m=+2.086682545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-multus-conf-dir\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-system-cni-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-etc-kubernetes\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-system-cni-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-modprobe-d\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-device-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kps\" (UniqueName: \"kubernetes.io/projected/08500cb6-32e2-49b8-bdaf-cad7b1599248-kube-api-access-d5kps\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672793    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-run\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-host\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.673776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672814    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/08500cb6-32e2-49b8-bdaf-cad7b1599248-device-dir\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672776    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-etc-kubernetes\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d035bba7-a661-4e73-ad2a-83643f775471-multus-daemon-config\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-run\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-tuned\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-os-release\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672939    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-multus\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-cnibin\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-os-release\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672987    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-systemd\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.673020    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-host\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.673027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d035bba7-a661-4e73-ad2a-83643f775471-host-var-lib-cni-multus\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.672999    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5c391c2f-95c7-491f-800c-2071ad9d4887-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.673041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c391c2f-95c7-491f-800c-2071ad9d4887-cnibin\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.673036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-systemd\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674269 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.674263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-tmp\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.674751 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.674558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/91f2519d-05ff-45e7-a6ea-9b929600c64e-etc-tuned\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.683978 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.683958    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:25.684074 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.683982    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:25.684074 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.683996    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:25.684074 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:25.684056    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:26.184039219 +0000 UTC m=+2.098095745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:25.684315 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.684091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzw4p\" (UniqueName: \"kubernetes.io/projected/91f2519d-05ff-45e7-a6ea-9b929600c64e-kube-api-access-gzw4p\") pod \"tuned-cdj97\" (UID: \"91f2519d-05ff-45e7-a6ea-9b929600c64e\") " pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.685184 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.685127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpv6v\" (UniqueName: \"kubernetes.io/projected/c549b894-3421-4622-8f97-d756354ab0f8-kube-api-access-lpv6v\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:25.685932 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.685912    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kps\" (UniqueName: \"kubernetes.io/projected/08500cb6-32e2-49b8-bdaf-cad7b1599248-kube-api-access-d5kps\") pod \"aws-ebs-csi-driver-node-w4qbz\" (UID: \"08500cb6-32e2-49b8-bdaf-cad7b1599248\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.686039 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.686023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqq5b\" (UniqueName: \"kubernetes.io/projected/5c391c2f-95c7-491f-800c-2071ad9d4887-kube-api-access-xqq5b\") pod \"multus-additional-cni-plugins-mmk5r\" (UID: \"5c391c2f-95c7-491f-800c-2071ad9d4887\") " pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.686125 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.686087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx4nc\" (UniqueName: \"kubernetes.io/projected/d035bba7-a661-4e73-ad2a-83643f775471-kube-api-access-xx4nc\") pod \"multus-kjkq8\" (UID: \"d035bba7-a661-4e73-ad2a-83643f775471\") " pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.776784 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.776738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kdn7c"
Jul 27 15:48:25.782848 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.782833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sst8w"
Jul 27 15:48:25.786068 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.786043    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1da3b24_14ce_4aa1_96ff_b045734336da.slice/crio-d5b0c484d8c40b031a064cc690df0172568d8c8e011505c5e8133f8c4a2e28e9 WatchSource:0}: Error finding container d5b0c484d8c40b031a064cc690df0172568d8c8e011505c5e8133f8c4a2e28e9: Status 404 returned error can't find the container with id d5b0c484d8c40b031a064cc690df0172568d8c8e011505c5e8133f8c4a2e28e9
Jul 27 15:48:25.790162 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.790129    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73411ea9_8cbf_44ed_b28d_b1dba00cd762.slice/crio-d3ff925e881d7cae0037d52b0be059581fa3f927631436e8dd4a575e5f617dbd WatchSource:0}: Error finding container d3ff925e881d7cae0037d52b0be059581fa3f927631436e8dd4a575e5f617dbd: Status 404 returned error can't find the container with id d3ff925e881d7cae0037d52b0be059581fa3f927631436e8dd4a575e5f617dbd
Jul 27 15:48:25.815963 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.815948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-76t4v"
Jul 27 15:48:25.822402 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.822382    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810304a7_7ea9_4638_a126_10f6765aabca.slice/crio-b1c6e2dfc1b853af67bf0c856c5470e80272e41b6929ff0908268f42a316ccd8 WatchSource:0}: Error finding container b1c6e2dfc1b853af67bf0c856c5470e80272e41b6929ff0908268f42a316ccd8: Status 404 returned error can't find the container with id b1c6e2dfc1b853af67bf0c856c5470e80272e41b6929ff0908268f42a316ccd8
Jul 27 15:48:25.828205 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.828189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:25.834536 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.834517    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955df176_6f69_4291_9ffa_25dd76c3f1ab.slice/crio-496e605ae363dd4b58668b1ca3bcefdc00b2f13f57e3ce20c73e88aaa4b79377 WatchSource:0}: Error finding container 496e605ae363dd4b58668b1ca3bcefdc00b2f13f57e3ce20c73e88aaa4b79377: Status 404 returned error can't find the container with id 496e605ae363dd4b58668b1ca3bcefdc00b2f13f57e3ce20c73e88aaa4b79377
Jul 27 15:48:25.847458 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.847442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:25.853720 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.853703    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0534f6_4899_4330_a551_9c446059d5e8.slice/crio-67990fd9a8e13a15e3572c633e521eb4dc9832a58853a537837791a724f6fcc1 WatchSource:0}: Error finding container 67990fd9a8e13a15e3572c633e521eb4dc9832a58853a537837791a724f6fcc1: Status 404 returned error can't find the container with id 67990fd9a8e13a15e3572c633e521eb4dc9832a58853a537837791a724f6fcc1
Jul 27 15:48:25.861609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.861595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cdj97"
Jul 27 15:48:25.867466 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.867415    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kjkq8"
Jul 27 15:48:25.869545 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.869477    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f2519d_05ff_45e7_a6ea_9b929600c64e.slice/crio-d4e5c7293fb64fa0c83268717c332286066040561a9b895d1470d8b8091925eb WatchSource:0}: Error finding container d4e5c7293fb64fa0c83268717c332286066040561a9b895d1470d8b8091925eb: Status 404 returned error can't find the container with id d4e5c7293fb64fa0c83268717c332286066040561a9b895d1470d8b8091925eb
Jul 27 15:48:25.874346 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.874328    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd035bba7_a661_4e73_ad2a_83643f775471.slice/crio-9d138ffa18d97bd5176b5f1011deedd490c97d5b773f000c9b66a63c0c16040c WatchSource:0}: Error finding container 9d138ffa18d97bd5176b5f1011deedd490c97d5b773f000c9b66a63c0c16040c: Status 404 returned error can't find the container with id 9d138ffa18d97bd5176b5f1011deedd490c97d5b773f000c9b66a63c0c16040c
Jul 27 15:48:25.897486 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.897459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz"
Jul 27 15:48:25.901672 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.901658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mmk5r"
Jul 27 15:48:25.906854 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.906834    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08500cb6_32e2_49b8_bdaf_cad7b1599248.slice/crio-c84b8dd4b0bf5ef0a505e4cfd5a1d92d6d23a834fece253ba943597dad724d0b WatchSource:0}: Error finding container c84b8dd4b0bf5ef0a505e4cfd5a1d92d6d23a834fece253ba943597dad724d0b: Status 404 returned error can't find the container with id c84b8dd4b0bf5ef0a505e4cfd5a1d92d6d23a834fece253ba943597dad724d0b
Jul 27 15:48:25.909254 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:25.909233    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c391c2f_95c7_491f_800c_2071ad9d4887.slice/crio-a72fec5325effbf63cacfac81c94d95778ecff71a26cbda8b42baa029d210d72 WatchSource:0}: Error finding container a72fec5325effbf63cacfac81c94d95778ecff71a26cbda8b42baa029d210d72: Status 404 returned error can't find the container with id a72fec5325effbf63cacfac81c94d95778ecff71a26cbda8b42baa029d210d72
Jul 27 15:48:25.960542 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:25.960523    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:26.076465 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.076395    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:26.076580 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.076536    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:26.076613 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.076595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:27.076575889 +0000 UTC m=+2.990632414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:26.177462 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.177429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:26.177858 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.177762    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:26.177858 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.177851    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:27.177829716 +0000 UTC m=+3.091886242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:26.216041 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:26.216013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a03fbe185acdf451816f7e23b5f888d.slice/crio-12852749881b3e7ada7b365e1833c54abcc64d5e098425aab646c3b1e1bca6fa WatchSource:0}: Error finding container 12852749881b3e7ada7b365e1833c54abcc64d5e098425aab646c3b1e1bca6fa: Status 404 returned error can't find the container with id 12852749881b3e7ada7b365e1833c54abcc64d5e098425aab646c3b1e1bca6fa
Jul 27 15:48:26.278425 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.278387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:26.278583 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.278538    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:26.278583 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.278556    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:26.278583 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.278567    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:26.278740 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.278634    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:27.278614652 +0000 UTC m=+3.192671199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:26.511785 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.511682    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 15:43:25 +0000 UTC" deadline="2028-04-26 06:19:28.77007536 +0000 UTC"
Jul 27 15:48:26.511785 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.511745    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15326h31m2.258334519s"
Jul 27 15:48:26.515261 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.515239    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:26.649360 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.649326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:26.649532 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:26.649467    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:26.661381 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.661347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerStarted","Data":"a72fec5325effbf63cacfac81c94d95778ecff71a26cbda8b42baa029d210d72"}
Jul 27 15:48:26.669235 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.669204    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kjkq8" event={"ID":"d035bba7-a661-4e73-ad2a-83643f775471","Type":"ContainerStarted","Data":"9d138ffa18d97bd5176b5f1011deedd490c97d5b773f000c9b66a63c0c16040c"}
Jul 27 15:48:26.674690 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.674634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdj97" event={"ID":"91f2519d-05ff-45e7-a6ea-9b929600c64e","Type":"ContainerStarted","Data":"d4e5c7293fb64fa0c83268717c332286066040561a9b895d1470d8b8091925eb"}
Jul 27 15:48:26.678862 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.678822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76t4v" event={"ID":"810304a7-7ea9-4638-a126-10f6765aabca","Type":"ContainerStarted","Data":"b1c6e2dfc1b853af67bf0c856c5470e80272e41b6929ff0908268f42a316ccd8"}
Jul 27 15:48:26.682409 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.682384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sst8w" event={"ID":"73411ea9-8cbf-44ed-b28d-b1dba00cd762","Type":"ContainerStarted","Data":"d3ff925e881d7cae0037d52b0be059581fa3f927631436e8dd4a575e5f617dbd"}
Jul 27 15:48:26.684026 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.684003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdn7c" event={"ID":"a1da3b24-14ce-4aa1-96ff-b045734336da","Type":"ContainerStarted","Data":"d5b0c484d8c40b031a064cc690df0172568d8c8e011505c5e8133f8c4a2e28e9"}
Jul 27 15:48:26.690385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.690362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal" event={"ID":"9a03fbe185acdf451816f7e23b5f888d","Type":"ContainerStarted","Data":"12852749881b3e7ada7b365e1833c54abcc64d5e098425aab646c3b1e1bca6fa"}
Jul 27 15:48:26.693711 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.693690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz" event={"ID":"08500cb6-32e2-49b8-bdaf-cad7b1599248","Type":"ContainerStarted","Data":"c84b8dd4b0bf5ef0a505e4cfd5a1d92d6d23a834fece253ba943597dad724d0b"}
Jul 27 15:48:26.697602 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.697548    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dp29f" event={"ID":"dd0534f6-4899-4330-a551-9c446059d5e8","Type":"ContainerStarted","Data":"67990fd9a8e13a15e3572c633e521eb4dc9832a58853a537837791a724f6fcc1"}
Jul 27 15:48:26.703943 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:26.703918    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"496e605ae363dd4b58668b1ca3bcefdc00b2f13f57e3ce20c73e88aaa4b79377"}
Jul 27 15:48:27.085478 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.085438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:27.085773 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.085598    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:27.085773 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.085659    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:29.085641265 +0000 UTC m=+4.999697808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:27.186456 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.186392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:27.186614 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.186536    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:27.186614 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.186596    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:29.186578276 +0000 UTC m=+5.100634815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:27.287651 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.287601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:27.287831 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.287792    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:27.287831 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.287811    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:27.287831 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.287825    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:27.287982 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.287900    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:29.287880316 +0000 UTC m=+5.201936857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:27.512540 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.512441    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 15:43:25 +0000 UTC" deadline="2028-05-02 06:42:23.40850212 +0000 UTC"
Jul 27 15:48:27.512540 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.512482    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15470h53m55.896024722s"
Jul 27 15:48:27.646776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.646749    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:27.646955 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.646877    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:27.647277 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:27.647258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:27.647377 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:27.647348    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:28.024095 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:28.024064    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 15:48:28.649913 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:28.649436    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:28.649913 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:28.649584    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:29.103302 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:29.103216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:29.103469 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.103381    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:29.103469 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.103447    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:33.103426732 +0000 UTC m=+9.017483270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:29.204062 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:29.203586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:29.204062 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.203778    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:29.204062 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.203838    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:33.20381912 +0000 UTC m=+9.117875660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:29.304078 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:29.304040    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:29.304289 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.304273    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:29.304383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.304295    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:29.304383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.304308    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:29.304383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.304363    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:33.304343689 +0000 UTC m=+9.218400235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:29.647156 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:29.647119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:29.647338 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.647248    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:29.647890 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:29.647669    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:29.647890 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:29.647789    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:30.648127 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:30.647622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:30.648127 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:30.647777    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:31.647186 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:31.647147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:31.647375 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:31.647147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:31.647375 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:31.647282    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:31.647487 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:31.647389    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:32.646636 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:32.646601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:32.647084 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:32.646754    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:33.138195 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:33.138130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:33.138383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.138275    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:33.138383 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.138343    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:41.138325218 +0000 UTC m=+17.052381744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:33.239200 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:33.239158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:33.239363 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.239339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:33.239429 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.239417    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:41.239395952 +0000 UTC m=+17.153452492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:33.340455 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:33.340414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:33.340645 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.340593    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:33.340645 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.340615    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:33.340645 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.340626    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:33.340798 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.340689    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:41.340671264 +0000 UTC m=+17.254727804 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:33.646515 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:33.646482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:33.646695 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.646614    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:33.646695 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:33.646659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:33.647158 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:33.646759    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:34.647851 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:34.647817    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:34.648310 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:34.647943    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:35.646552 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:35.646515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:35.646733 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:35.646649    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:35.646733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:35.646699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:35.646844 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:35.646810    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:36.646995 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:36.646968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:36.647353 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:36.647075    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:37.646363 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:37.646332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:37.646564 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:37.646459    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:37.646564 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:37.646519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:37.646677 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:37.646621    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:38.646599 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:38.646559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:38.647049 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:38.646714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:39.646506 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:39.646468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:39.646661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:39.646475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:39.646661 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:39.646603    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:39.647046 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:39.646665    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:40.646752 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:40.646712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:40.647153 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:40.646863    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:41.202315 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:41.202280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:41.202503 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.202459    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:41.202565 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.202535    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:57.202513592 +0000 UTC m=+33.116570152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:41.303354 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:41.303323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:41.303529 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.303507    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:41.303598 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.303584    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:57.303561048 +0000 UTC m=+33.217617591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:41.403887 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:41.403857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:41.404071 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.404033    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:41.404071 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.404063    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:41.404220 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.404076    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:41.404220 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.404149    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:48:57.404128809 +0000 UTC m=+33.318185336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:41.646831 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:41.646757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:41.647350 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.646863    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:41.647350 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:41.646924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:41.647350 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:41.647035    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:42.646841 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:42.646597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:42.647303 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:42.646960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:43.647196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:43.647164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:43.647622 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:43.647164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:43.647622 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:43.647274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:43.647622 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:43.647387    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:44.647182 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:44.647145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:44.647357 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:44.647277    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:45.647176 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.646968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:45.647351 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.646992    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:45.647351 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:45.647288    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:45.648021 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:45.647354    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:45.743421 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.743388    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kjkq8" event={"ID":"d035bba7-a661-4e73-ad2a-83643f775471","Type":"ContainerStarted","Data":"1c7ab10b446fb4b2bf7f5966fa0064e38717d6ac1f0cb4ea148b706d5e18ca87"}
Jul 27 15:48:45.744843 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.744820    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cdj97" event={"ID":"91f2519d-05ff-45e7-a6ea-9b929600c64e","Type":"ContainerStarted","Data":"2186d88e1b3c59e81f3feb2bdbbabd408b3dfdda961e921aff31e3de648b4e1b"}
Jul 27 15:48:45.746497 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.746471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sst8w" event={"ID":"73411ea9-8cbf-44ed-b28d-b1dba00cd762","Type":"ContainerStarted","Data":"12c035b56d931484a159972abac7374ba7e0fc53c044459b616c2e9c20ded2d0"}
Jul 27 15:48:45.748213 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.748189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kdn7c" event={"ID":"a1da3b24-14ce-4aa1-96ff-b045734336da","Type":"ContainerStarted","Data":"e64d07b477d85e73f41ddafb03973deb196f34604a6f168afc335d091cb724ce"}
Jul 27 15:48:45.749529 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.749509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal" event={"ID":"9a03fbe185acdf451816f7e23b5f888d","Type":"ContainerStarted","Data":"27e679155cd69cc8b7e5ea538e944b76cad633ffaebf5bdd4762767aa0818a2a"}
Jul 27 15:48:45.750723 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.750704    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz" event={"ID":"08500cb6-32e2-49b8-bdaf-cad7b1599248","Type":"ContainerStarted","Data":"e3008ed3fbf4d87ca1fe45cb3b003d8f545ecf049efe5285c06e97cd8510c387"}
Jul 27 15:48:45.751906 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.751890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dp29f" event={"ID":"dd0534f6-4899-4330-a551-9c446059d5e8","Type":"ContainerStarted","Data":"12e338d22e9d894a8a3f984a5024bc13311d60b355759884ab0923d1ebdd2ae3"}
Jul 27 15:48:45.753662 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.753645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"edc7c6c61562be17ff629f3259d284549d16f85defbb7a30bd582e8083878071"}
Jul 27 15:48:45.753723 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.753703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"3fb463022a7e379a1480b53e5b7e7848978d2ac28d9291a05c660cb3d790b1c9"}
Jul 27 15:48:45.753755 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.753740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"3bf0ec7436c17103cef6691e6f80da7238adb571faeeb3dd0584b10f13d90db5"}
Jul 27 15:48:45.754942 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.754924    2576 generic.go:356] "Generic (PLEG): container finished" podID="d6570cf7faba16b53fa79128551a488d" containerID="9a679a0c437730b3be5c9a2fed0a80ebdf0fc248eecda2b74e1f9d1049c5b210" exitCode=0
Jul 27 15:48:45.755050 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.754985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal" event={"ID":"d6570cf7faba16b53fa79128551a488d","Type":"ContainerDied","Data":"9a679a0c437730b3be5c9a2fed0a80ebdf0fc248eecda2b74e1f9d1049c5b210"}
Jul 27 15:48:45.756431 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.756413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerStarted","Data":"de6d1504389f2fef70936d16f9af8af14566f575671e5b19363ebe9c326d8028"}
Jul 27 15:48:45.762246 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.762207    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kjkq8" podStartSLOduration=2.540538873 podStartE2EDuration="21.762193752s" totalImagesPullingTime="19.221654879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.762143068 +0000 UTC m=+21.676199616" watchObservedRunningTime="2026-07-27 15:48:45.762193752 +0000 UTC m=+21.676250299"
Jul 27 15:48:45.775901 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.775860    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kdn7c" podStartSLOduration=2.501802058 podStartE2EDuration="21.775841447s" totalImagesPullingTime="19.274039389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.775722191 +0000 UTC m=+21.689778738" watchObservedRunningTime="2026-07-27 15:48:45.775841447 +0000 UTC m=+21.689897997"
Jul 27 15:48:45.794760 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.794722    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cdj97" podStartSLOduration=2.5909775489999998 podStartE2EDuration="21.794705997s" totalImagesPullingTime="19.203728448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.794238566 +0000 UTC m=+21.708295117" watchObservedRunningTime="2026-07-27 15:48:45.794705997 +0000 UTC m=+21.708762546"
Jul 27 15:48:45.830558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.830519    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sst8w" podStartSLOduration=10.84671715 podStartE2EDuration="21.830506543s" totalImagesPullingTime="10.983789393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.830045411 +0000 UTC m=+21.744101959" watchObservedRunningTime="2026-07-27 15:48:45.830506543 +0000 UTC m=+21.744563090"
Jul 27 15:48:45.850466 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:45.850410    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dp29f" podStartSLOduration=2.676719499 podStartE2EDuration="21.850390732s" totalImagesPullingTime="19.173671233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.850288575 +0000 UTC m=+21.764345118" watchObservedRunningTime="2026-07-27 15:48:45.850390732 +0000 UTC m=+21.764447283"
Jul 27 15:48:46.646825 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.646617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:46.647011 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:46.646941    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:46.762784 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.762749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"b2ae142af784a42c9785c57009b1e270bc9da57c3445abd7f3bad6419aa514f0"}
Jul 27 15:48:46.763408 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.762861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"733bb612b939d6c6838be0685b216b418b0e1d2b7415d7a809b69fca5223430f"}
Jul 27 15:48:46.763408 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.762946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"4885e648db7d1acb16c14ecae1df7ffa9016a552a7e682a7cf23fe8edd89e9a4"}
Jul 27 15:48:46.764699 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.764671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal" event={"ID":"d6570cf7faba16b53fa79128551a488d","Type":"ContainerStarted","Data":"b0962e77e50bd3344b4685b3bcf1bcf50307af4488194554de4d2280db8fdf45"}
Jul 27 15:48:46.766394 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.766371    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="de6d1504389f2fef70936d16f9af8af14566f575671e5b19363ebe9c326d8028" exitCode=0
Jul 27 15:48:46.766506 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.766454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"de6d1504389f2fef70936d16f9af8af14566f575671e5b19363ebe9c326d8028"}
Jul 27 15:48:46.768028 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.767913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-76t4v" event={"ID":"810304a7-7ea9-4638-a126-10f6765aabca","Type":"ContainerStarted","Data":"a94ba60b1ab483c44a4e431bb0017739569b7b6bdce308436a4b38dc71017009"}
Jul 27 15:48:46.780847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.780806    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-171.ec2.internal" podStartSLOduration=21.780792957 podStartE2EDuration="21.780792957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:45.888685657 +0000 UTC m=+21.802742205" watchObservedRunningTime="2026-07-27 15:48:46.780792957 +0000 UTC m=+22.694849518"
Jul 27 15:48:46.781240 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.781210    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-171.ec2.internal" podStartSLOduration=21.781203324 podStartE2EDuration="21.781203324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:46.780851022 +0000 UTC m=+22.694907569" watchObservedRunningTime="2026-07-27 15:48:46.781203324 +0000 UTC m=+22.695259871"
Jul 27 15:48:46.795605 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:46.795568    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-76t4v" podStartSLOduration=3.546625646 podStartE2EDuration="22.795556349s" totalImagesPullingTime="19.248930703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:46.795400851 +0000 UTC m=+22.709457401" watchObservedRunningTime="2026-07-27 15:48:46.795556349 +0000 UTC m=+22.709612897"
Jul 27 15:48:47.646253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:47.646219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:47.646253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:47.646263    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:47.646700 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:47.646356    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:47.646700 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:47.646510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:48.120084 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.120055    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 15:48:48.538213 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.538098    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T15:48:48.120251511Z","UUID":"e54e0910-9541-4a6c-af90-91abe860861d","Handler":null,"Name":"","Endpoint":""}
Jul 27 15:48:48.539593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.539555    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 15:48:48.539593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.539598    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 15:48:48.647225 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.647193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:48.647393 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:48.647327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:48.773747 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.773712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz" event={"ID":"08500cb6-32e2-49b8-bdaf-cad7b1599248","Type":"ContainerStarted","Data":"54e2af8b7672f2436d37aee6e9615281b921e34e8c4138cba5a606a207f4fbea"}
Jul 27 15:48:48.777319 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:48.777293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"68e7fe2a88d2d1086fbd7269814faefb3f10e8a3f64c4d027b5a309bd6ebd8c0"}
Jul 27 15:48:49.245279 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.245248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:49.246286 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.246270    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:49.647018 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.646947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:49.647018 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.646978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:49.647240 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:49.647053    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:49.647240 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:49.647224    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:49.779342 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.779316    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:49.779811 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:49.779787    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dp29f"
Jul 27 15:48:50.646470 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.646195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:50.646904 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:50.646551    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:50.784411 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.784381    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" event={"ID":"955df176-6f69-4291-9ffa-25dd76c3f1ab","Type":"ContainerStarted","Data":"7cb3a7c254b5bd778434d630f704af9de172e3408d605449a20cc49e1913ada4"}
Jul 27 15:48:50.784605 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.784589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:50.784678 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.784662    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:50.784734 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.784711    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:50.786975 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.786945    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="296914b4f7749e078a752270ae12663d38dc249563bd322d138037f5c1e1168c" exitCode=0
Jul 27 15:48:50.787149 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.787095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"296914b4f7749e078a752270ae12663d38dc249563bd322d138037f5c1e1168c"}
Jul 27 15:48:50.801385 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.801321    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:50.801787 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.801768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:48:50.811290 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:50.811252    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2" podStartSLOduration=7.239200046 podStartE2EDuration="26.811240659s" totalImagesPullingTime="19.572040613s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:50.810904473 +0000 UTC m=+26.724961021" watchObservedRunningTime="2026-07-27 15:48:50.811240659 +0000 UTC m=+26.725297207"
Jul 27 15:48:51.646513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:51.646476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:51.647313 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:51.646612    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:51.647313 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:51.646661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:51.647313 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:51.646829    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:51.792756 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:51.792325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz" event={"ID":"08500cb6-32e2-49b8-bdaf-cad7b1599248","Type":"ContainerStarted","Data":"aafc27957a9adef984a95ed25ab46b89accc2c9ad890c0e6d882ef17923abdda"}
Jul 27 15:48:52.432517 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.432261    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w4qbz" podStartSLOduration=4.444881656 podStartE2EDuration="28.432238424s" totalImagesPullingTime="23.987356768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:48:51.812090215 +0000 UTC m=+27.726146783" watchObservedRunningTime="2026-07-27 15:48:52.432238424 +0000 UTC m=+28.346294973"
Jul 27 15:48:52.432794 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.432775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-49g5r"]
Jul 27 15:48:52.432947 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.432932    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:52.433054 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:52.433036    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:52.435273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.435246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qd4fh"]
Jul 27 15:48:52.435379 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.435357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:52.435502 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:52.435469    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:52.435813 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.435794    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2lzpz"]
Jul 27 15:48:52.435900 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.435889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:52.435980 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:52.435963    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:52.796409 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.796374    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="8bc18f81a028df8886b124434a39b3b2b808b6c2ee3df4d6671f59f06dae3e08" exitCode=0
Jul 27 15:48:52.796993 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:52.796428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"8bc18f81a028df8886b124434a39b3b2b808b6c2ee3df4d6671f59f06dae3e08"}
Jul 27 15:48:53.646157 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:53.646130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:53.646341 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:53.646229    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:54.647259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:54.647234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:54.647668 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:54.647314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:54.647668 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:54.647349    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:54.647668 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:54.647438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:54.802541 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:54.802515    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="4d95e3d8c2eaf79d79b9c10802608e42466b18995011915086bc71f860baa7bd" exitCode=0
Jul 27 15:48:54.802643 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:54.802568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"4d95e3d8c2eaf79d79b9c10802608e42466b18995011915086bc71f860baa7bd"}
Jul 27 15:48:55.647115 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:55.647072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:55.647301 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:55.647221    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:56.650551 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:56.650521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:56.650999 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:56.650646    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qd4fh" podUID="f8a436aa-e355-41f9-ae1b-8caa9b42dcf8"
Jul 27 15:48:56.650999 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:56.650694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:56.650999 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:56.650825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:48:57.227463 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.227426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:57.227647 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.227547    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:57.227647 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.227615    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret podName:f8a436aa-e355-41f9-ae1b-8caa9b42dcf8 nodeName:}" failed. No retries permitted until 2026-07-27 15:49:29.227600011 +0000 UTC m=+65.141656557 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret") pod "global-pull-secret-syncer-qd4fh" (UID: "f8a436aa-e355-41f9-ae1b-8caa9b42dcf8") : object "kube-system"/"original-pull-secret" not registered
Jul 27 15:48:57.328410 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.328378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:57.328578 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.328523    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:57.328628 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.328587    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:49:29.328567842 +0000 UTC m=+65.242624391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 15:48:57.429461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.429401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:57.429657 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.429593    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 15:48:57.429657 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.429619    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 15:48:57.429657 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.429631    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bgq92 for pod openshift-network-diagnostics/network-check-target-49g5r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:57.429808 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.429692    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92 podName:6ac464fc-2b68-4160-9e6d-fea988fcce37 nodeName:}" failed. No retries permitted until 2026-07-27 15:49:29.429672519 +0000 UTC m=+65.343729059 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgq92" (UniqueName: "kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92") pod "network-check-target-49g5r" (UID: "6ac464fc-2b68-4160-9e6d-fea988fcce37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 15:48:57.646364 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.646281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:57.646512 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:57.646427    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-49g5r" podUID="6ac464fc-2b68-4160-9e6d-fea988fcce37"
Jul 27 15:48:57.957992 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.957964    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-171.ec2.internal" event="NodeReady"
Jul 27 15:48:57.958401 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.958098    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 15:48:57.999153 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:57.999123    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mbxq7"]
Jul 27 15:48:58.001950 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.001931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.004598 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.004576    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 15:48:58.004728 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.004670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 15:48:58.004728 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.004693    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-748tg\""
Jul 27 15:48:58.005620 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.005600    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-74bjz"]
Jul 27 15:48:58.007443 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.007427    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.010486 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.010078    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 15:48:58.010486 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.010336    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 15:48:58.010486 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.010443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 15:48:58.010711 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.010560    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-54wz2\""
Jul 27 15:48:58.011284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.011265    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbxq7"]
Jul 27 15:48:58.016601 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.016567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74bjz"]
Jul 27 15:48:58.134670 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm949\" (UniqueName: \"kubernetes.io/projected/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-kube-api-access-rm949\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.134670 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-config-volume\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.134898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.134898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-tmp-dir\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.134898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9881139c-b0ec-4a87-8c39-73f5c23964a3-cert\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.134898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.134829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zb2\" (UniqueName: \"kubernetes.io/projected/9881139c-b0ec-4a87-8c39-73f5c23964a3-kube-api-access-v9zb2\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.235504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm949\" (UniqueName: \"kubernetes.io/projected/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-kube-api-access-rm949\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.235504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-config-volume\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.235504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.235504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-tmp-dir\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.235819 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235570    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9881139c-b0ec-4a87-8c39-73f5c23964a3-cert\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.235819 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zb2\" (UniqueName: \"kubernetes.io/projected/9881139c-b0ec-4a87-8c39-73f5c23964a3-kube-api-access-v9zb2\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.235819 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:58.235617    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:48:58.235819 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:58.235690    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:48:58.735670654 +0000 UTC m=+34.649727195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:48:58.236014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.235834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-tmp-dir\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.236242 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.236225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-config-volume\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.240758 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.240632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9881139c-b0ec-4a87-8c39-73f5c23964a3-cert\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.245817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.245762    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm949\" (UniqueName: \"kubernetes.io/projected/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-kube-api-access-rm949\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.245817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.245769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zb2\" (UniqueName: \"kubernetes.io/projected/9881139c-b0ec-4a87-8c39-73f5c23964a3-kube-api-access-v9zb2\") pod \"ingress-canary-74bjz\" (UID: \"9881139c-b0ec-4a87-8c39-73f5c23964a3\") " pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.322783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.322751    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-74bjz"
Jul 27 15:48:58.649796 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.649773    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:48:58.649884 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.649813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:48:58.653480 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.653458    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92bc\""
Jul 27 15:48:58.653566 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.653465    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 15:48:58.653566 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.653467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 15:48:58.739304 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.739274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:58.739435 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:58.739414    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:48:58.739507 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:58.739488    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:48:59.739468807 +0000 UTC m=+35.653525335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:48:58.741539 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.741512    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-74bjz"]
Jul 27 15:48:58.744489 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:48:58.744464    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9881139c_b0ec_4a87_8c39_73f5c23964a3.slice/crio-99afee06528bde29de60cd04850ab1c5489c47b023147beca820ba3827b4fc51 WatchSource:0}: Error finding container 99afee06528bde29de60cd04850ab1c5489c47b023147beca820ba3827b4fc51: Status 404 returned error can't find the container with id 99afee06528bde29de60cd04850ab1c5489c47b023147beca820ba3827b4fc51
Jul 27 15:48:58.813642 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.813610    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74bjz" event={"ID":"9881139c-b0ec-4a87-8c39-73f5c23964a3","Type":"ContainerStarted","Data":"99afee06528bde29de60cd04850ab1c5489c47b023147beca820ba3827b4fc51"}
Jul 27 15:48:58.816464 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:58.816443    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerStarted","Data":"7405f1f3ececdf636aff800c0b01636176f60ed93d716b5d419208d09c449837"}
Jul 27 15:48:59.647276 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.647241    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:48:59.650450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.650426    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 15:48:59.651389 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.651369    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 15:48:59.651389 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.651384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lcv26\""
Jul 27 15:48:59.748089 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.748060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:48:59.748215 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:59.748191    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:48:59.748274 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:48:59.748268    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:49:01.748249893 +0000 UTC m=+37.662306418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:48:59.823403 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.823038    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="7405f1f3ececdf636aff800c0b01636176f60ed93d716b5d419208d09c449837" exitCode=0
Jul 27 15:48:59.823403 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:48:59.823130    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"7405f1f3ececdf636aff800c0b01636176f60ed93d716b5d419208d09c449837"}
Jul 27 15:49:00.829358 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:00.829325    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c391c2f-95c7-491f-800c-2071ad9d4887" containerID="a1e51d1bb659e64ccd745249b8e5f96e9d17d53fe2baf0cb61ed08c9174e86f5" exitCode=0
Jul 27 15:49:00.829813 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:00.829386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerDied","Data":"a1e51d1bb659e64ccd745249b8e5f96e9d17d53fe2baf0cb61ed08c9174e86f5"}
Jul 27 15:49:01.763639 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:01.763450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:49:01.763784 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:01.763600    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:49:01.763784 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:01.763724    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:49:05.763708464 +0000 UTC m=+41.677764990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:49:01.835518 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:01.835478    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" event={"ID":"5c391c2f-95c7-491f-800c-2071ad9d4887","Type":"ContainerStarted","Data":"4f51dec4c41b6285e0e757c8b8781a2da2eb50b9110a93c6b34c95d01011619e"}
Jul 27 15:49:01.836905 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:01.836885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-74bjz" event={"ID":"9881139c-b0ec-4a87-8c39-73f5c23964a3","Type":"ContainerStarted","Data":"e53c72caeb08cacf97921a439b92b7d65a726b5ff10d71e60b88bcb6d362cd67"}
Jul 27 15:49:01.861609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:01.861571    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mmk5r" podStartSLOduration=8.290924109 podStartE2EDuration="37.861557242s" totalImagesPullingTime="29.570633133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:49:01.859963655 +0000 UTC m=+37.774020202" watchObservedRunningTime="2026-07-27 15:49:01.861557242 +0000 UTC m=+37.775613790"
Jul 27 15:49:01.877620 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:01.877582    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-74bjz" podStartSLOduration=2.508153748 podStartE2EDuration="4.87756937s" totalImagesPullingTime="2.369415622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:49:01.876265208 +0000 UTC m=+37.790321767" watchObservedRunningTime="2026-07-27 15:49:01.87756937 +0000 UTC m=+37.791625917"
Jul 27 15:49:05.794543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:05.794505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:49:05.794927 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:05.794632    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:49:05.794927 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:05.794688    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:49:13.794671728 +0000 UTC m=+49.708728257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:49:13.849871 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:13.849832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:49:13.850357 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:13.849952    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:49:13.850357 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:13.850003    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:49:29.849990248 +0000 UTC m=+65.764046774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:49:22.815783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:22.815755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qh8s2"
Jul 27 15:49:29.254524 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.254472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:49:29.257601 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.257571    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 15:49:29.268264 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.268247    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f8a436aa-e355-41f9-ae1b-8caa9b42dcf8-original-pull-secret\") pod \"global-pull-secret-syncer-qd4fh\" (UID: \"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8\") " pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:49:29.355589 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.355560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:49:29.358379 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.358360    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 15:49:29.366310 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:29.366290    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 15:49:29.366377 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:29.366344    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:50:33.366326624 +0000 UTC m=+129.280383152 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : secret "metrics-daemon-secret" not found
Jul 27 15:49:29.456552 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.456528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:49:29.459225 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.459209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 15:49:29.469713 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.469692    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 15:49:29.479827 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.479808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgq92\" (UniqueName: \"kubernetes.io/projected/6ac464fc-2b68-4160-9e6d-fea988fcce37-kube-api-access-bgq92\") pod \"network-check-target-49g5r\" (UID: \"6ac464fc-2b68-4160-9e6d-fea988fcce37\") " pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:49:29.561191 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.561134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qd4fh"
Jul 27 15:49:29.659765 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.659741    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lcv26\""
Jul 27 15:49:29.667741 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.667722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:49:29.676366 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.676313    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qd4fh"]
Jul 27 15:49:29.680713 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:49:29.680689    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a436aa_e355_41f9_ae1b_8caa9b42dcf8.slice/crio-a82eaa5d4d98c0bc57ab40eb12e36efd64d6107e3b130ff27bd7e99acc3a1761 WatchSource:0}: Error finding container a82eaa5d4d98c0bc57ab40eb12e36efd64d6107e3b130ff27bd7e99acc3a1761: Status 404 returned error can't find the container with id a82eaa5d4d98c0bc57ab40eb12e36efd64d6107e3b130ff27bd7e99acc3a1761
Jul 27 15:49:29.793718 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.793687    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-49g5r"]
Jul 27 15:49:29.796238 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:49:29.796212    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac464fc_2b68_4160_9e6d_fea988fcce37.slice/crio-d640a0522648fe197d88212825b98c2899f74c21c32f5f558cb5e176eb8563c3 WatchSource:0}: Error finding container d640a0522648fe197d88212825b98c2899f74c21c32f5f558cb5e176eb8563c3: Status 404 returned error can't find the container with id d640a0522648fe197d88212825b98c2899f74c21c32f5f558cb5e176eb8563c3
Jul 27 15:49:29.859080 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.859031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:49:29.859179 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:29.859149    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:49:29.859223 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:49:29.859196    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:50:01.859183195 +0000 UTC m=+97.773239721 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:49:29.902910 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.902883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-49g5r" event={"ID":"6ac464fc-2b68-4160-9e6d-fea988fcce37","Type":"ContainerStarted","Data":"d640a0522648fe197d88212825b98c2899f74c21c32f5f558cb5e176eb8563c3"}
Jul 27 15:49:29.903917 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:29.903895    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qd4fh" event={"ID":"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8","Type":"ContainerStarted","Data":"a82eaa5d4d98c0bc57ab40eb12e36efd64d6107e3b130ff27bd7e99acc3a1761"}
Jul 27 15:49:34.917765 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:34.917726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-49g5r" event={"ID":"6ac464fc-2b68-4160-9e6d-fea988fcce37","Type":"ContainerStarted","Data":"b623566962688e6665f62d320df45c9d8cf5cad7466524260777446686259d05"}
Jul 27 15:49:34.918291 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:34.917801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:49:34.919146 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:34.919127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qd4fh" event={"ID":"f8a436aa-e355-41f9-ae1b-8caa9b42dcf8","Type":"ContainerStarted","Data":"43038998dee78c6cf259b19383ef7fc876931e33772252bed489ffb9d5b1bbb8"}
Jul 27 15:49:34.937079 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:34.937022    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-49g5r" podStartSLOduration=66.440330658 podStartE2EDuration="1m10.937007532s" totalImagesPullingTime="4.496676874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:49:34.936961129 +0000 UTC m=+70.851017678" watchObservedRunningTime="2026-07-27 15:49:34.937007532 +0000 UTC m=+70.851064081"
Jul 27 15:49:34.953031 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:49:34.952990    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qd4fh" podStartSLOduration=66.33700909 podStartE2EDuration="1m10.952977061s" totalImagesPullingTime="4.615967971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:49:34.952338782 +0000 UTC m=+70.866395332" watchObservedRunningTime="2026-07-27 15:49:34.952977061 +0000 UTC m=+70.867033609"
Jul 27 15:50:01.880453 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:50:01.880319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:50:01.880840 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:50:01.880508    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:50:01.880840 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:50:01.880573    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:51:05.880558132 +0000 UTC m=+161.794614658 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:50:05.926253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:50:05.926223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-49g5r"
Jul 27 15:50:33.395719 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:50:33.395678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:50:33.396285 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:50:33.395806    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 15:50:33.396285 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:50:33.395884    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs podName:c549b894-3421-4622-8f97-d756354ab0f8 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:35.395863793 +0000 UTC m=+251.309920320 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs") pod "network-metrics-daemon-2lzpz" (UID: "c549b894-3421-4622-8f97-d756354ab0f8") : secret "metrics-daemon-secret" not found
Jul 27 15:51:01.017231 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:01.017187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mbxq7" podUID="dc31d968-2a9a-4cb6-8ad5-3b40b9453eac"
Jul 27 15:51:01.117904 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:01.117876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:51:01.667861 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:01.667811    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2lzpz" podUID="c549b894-3421-4622-8f97-d756354ab0f8"
Jul 27 15:51:05.908773 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:05.908728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:51:05.909188 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:05.908879    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 15:51:05.909188 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:05.908935    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls podName:dc31d968-2a9a-4cb6-8ad5-3b40b9453eac nodeName:}" failed. No retries permitted until 2026-07-27 15:53:07.908920497 +0000 UTC m=+283.822977027 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls") pod "dns-default-mbxq7" (UID: "dc31d968-2a9a-4cb6-8ad5-3b40b9453eac") : secret "dns-default-metrics-tls" not found
Jul 27 15:51:14.647607 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:14.647571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:51:34.499136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.499041    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h7n4w"]
Jul 27 15:51:34.500715 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.500699    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6848cdb7df-qbb29"]
Jul 27 15:51:34.500833 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.500815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.502421 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.502404    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.504762 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.504738    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 15:51:34.504869 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.504819    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 15:51:34.505260 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505247    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 15:51:34.505260 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505251    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 15:51:34.505484 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505470    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dhx4k\""
Jul 27 15:51:34.505705 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505693    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 27 15:51:34.505877 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 15:51:34.505987 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.505971    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 27 15:51:34.506465 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.506447    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 15:51:34.506923 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.506909    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 15:51:34.506995 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.506932    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-64sjr\""
Jul 27 15:51:34.506995 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.506940    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 15:51:34.511715 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.511640    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 27 15:51:34.520621 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.520600    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h7n4w"]
Jul 27 15:51:34.527260 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.527235    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6848cdb7df-qbb29"]
Jul 27 15:51:34.587683 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.587661    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"]
Jul 27 15:51:34.589509 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.589491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"
Jul 27 15:51:34.590425 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.590405    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qq9gn"]
Jul 27 15:51:34.594264 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.594244    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rqhj8\""
Jul 27 15:51:34.595472 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.595823 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-snapshots\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.595896 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wv2\" (UniqueName: \"kubernetes.io/projected/395f36f7-8a82-499c-b1e8-f698f52f2594-kube-api-access-79wv2\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.595896 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.595896 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595875    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.596093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psj8z\" (UniqueName: \"kubernetes.io/projected/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-kube-api-access-psj8z\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.596093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.595996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-serving-cert\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.596093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.596037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.596093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.596063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-tmp\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.596093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.596091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-stats-auth\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.596276 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.596149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-service-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.596276 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.596198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-default-certificate\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.598558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.598528    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 15:51:34.598654 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.598530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 15:51:34.598654 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.598533    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 15:51:34.599842 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.599729    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"]
Jul 27 15:51:34.599842 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.599824    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2ndz4\""
Jul 27 15:51:34.600169 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.600147    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 15:51:34.601606 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.601588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"
Jul 27 15:51:34.603377 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.603354    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 15:51:34.603643 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.603624    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"]
Jul 27 15:51:34.608078 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.608056    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 15:51:34.608199 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.608083    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 15:51:34.608199 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.608154    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9mhzj\""
Jul 27 15:51:34.619547 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.619529    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"]
Jul 27 15:51:34.620293 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.620266    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qq9gn"]
Jul 27 15:51:34.641584 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.641560    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"]
Jul 27 15:51:34.643333 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.643316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.645985 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.645967    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9hr5f\""
Jul 27 15:51:34.646147 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.646132    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 15:51:34.646239 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.646223    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 15:51:34.646305 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.646283    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 27 15:51:34.646513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.646492    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 15:51:34.659977 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.659957    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"]
Jul 27 15:51:34.697296 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.697402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-tmp\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.697402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-stats-auth\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.697402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-service-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.697562 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-default-certificate\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.697562 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:34.697439    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:34.697562 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:34.697509    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:35.197488472 +0000 UTC m=+191.111545016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:34.697562 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-snapshots\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697566    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98e29607-f7f5-4282-98d9-b2dd928e34d5-serving-cert\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-trusted-ca\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557b7861-bc40-4685-9a4c-1c031bf6d492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7q62\" (UniqueName: \"kubernetes.io/projected/98e29607-f7f5-4282-98d9-b2dd928e34d5-kube-api-access-p7q62\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lql\" (UniqueName: \"kubernetes.io/projected/557b7861-bc40-4685-9a4c-1c031bf6d492-kube-api-access-k7lql\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wv2\" (UniqueName: \"kubernetes.io/projected/395f36f7-8a82-499c-b1e8-f698f52f2594-kube-api-access-79wv2\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.697783 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5m5f\" (UniqueName: \"kubernetes.io/projected/ea31c5ca-0c82-4d56-b52a-011f3de07eaa-kube-api-access-k5m5f\") pod \"volume-data-source-validator-549c6d54cf-d65gl\" (UID: \"ea31c5ca-0c82-4d56-b52a-011f3de07eaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697803    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmft\" (UniqueName: \"kubernetes.io/projected/3edd01d4-7f69-42d5-a9aa-03dea15f08e5-kube-api-access-tfmft\") pod \"network-check-source-7d88cb55d8-2tgtj\" (UID: \"3edd01d4-7f69-42d5-a9aa-03dea15f08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psj8z\" (UniqueName: \"kubernetes.io/projected/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-kube-api-access-psj8z\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-config\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-serving-cert\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:34.697950    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:35.197930753 +0000 UTC m=+191.111987284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-tmp\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.698232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.697986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-service-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.698582 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.698260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-snapshots\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.698710 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.698692    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.699882 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.699856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-default-certificate\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.699966 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.699862    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-stats-auth\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.700079 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.700061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-serving-cert\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.706647 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.706629    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"]
Jul 27 15:51:34.709011 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.708997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.715728 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.715704    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 15:51:34.716176 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.716163    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psj8z\" (UniqueName: \"kubernetes.io/projected/a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa-kube-api-access-psj8z\") pod \"insights-operator-5885fdb759-h7n4w\" (UID: \"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa\") " pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.716226 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.716179    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-t6pcg\""
Jul 27 15:51:34.716511 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.716487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 15:51:34.718506 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.718490    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wv2\" (UniqueName: \"kubernetes.io/projected/395f36f7-8a82-499c-b1e8-f698f52f2594-kube-api-access-79wv2\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:34.730771 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.730748    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"]
Jul 27 15:51:34.799148 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5m5f\" (UniqueName: \"kubernetes.io/projected/ea31c5ca-0c82-4d56-b52a-011f3de07eaa-kube-api-access-k5m5f\") pod \"volume-data-source-validator-549c6d54cf-d65gl\" (UID: \"ea31c5ca-0c82-4d56-b52a-011f3de07eaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"
Jul 27 15:51:34.799148 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmft\" (UniqueName: \"kubernetes.io/projected/3edd01d4-7f69-42d5-a9aa-03dea15f08e5-kube-api-access-tfmft\") pod \"network-check-source-7d88cb55d8-2tgtj\" (UID: \"3edd01d4-7f69-42d5-a9aa-03dea15f08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"
Jul 27 15:51:34.799282 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-trusted-ca\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.799282 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557b7861-bc40-4685-9a4c-1c031bf6d492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.799282 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799221    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lql\" (UniqueName: \"kubernetes.io/projected/557b7861-bc40-4685-9a4c-1c031bf6d492-kube-api-access-k7lql\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.799375 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-config\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.799375 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.799438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98e29607-f7f5-4282-98d9-b2dd928e34d5-serving-cert\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.799438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799426    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7q62\" (UniqueName: \"kubernetes.io/projected/98e29607-f7f5-4282-98d9-b2dd928e34d5-kube-api-access-p7q62\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.799522 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.799924 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.799898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-config\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.800196 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.800177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/98e29607-f7f5-4282-98d9-b2dd928e34d5-trusted-ca\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.801499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.801482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/557b7861-bc40-4685-9a4c-1c031bf6d492-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.801499 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.801496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98e29607-f7f5-4282-98d9-b2dd928e34d5-serving-cert\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.809968 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.809946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lql\" (UniqueName: \"kubernetes.io/projected/557b7861-bc40-4685-9a4c-1c031bf6d492-kube-api-access-k7lql\") pod \"managed-serviceaccount-addon-agent-666b7c558b-j4nks\" (UID: \"557b7861-bc40-4685-9a4c-1c031bf6d492\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:34.810550 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.810534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-h7n4w"
Jul 27 15:51:34.811073 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.811056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7q62\" (UniqueName: \"kubernetes.io/projected/98e29607-f7f5-4282-98d9-b2dd928e34d5-kube-api-access-p7q62\") pod \"console-operator-584dcb6b54-qq9gn\" (UID: \"98e29607-f7f5-4282-98d9-b2dd928e34d5\") " pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.812063 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.812042    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmft\" (UniqueName: \"kubernetes.io/projected/3edd01d4-7f69-42d5-a9aa-03dea15f08e5-kube-api-access-tfmft\") pod \"network-check-source-7d88cb55d8-2tgtj\" (UID: \"3edd01d4-7f69-42d5-a9aa-03dea15f08e5\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"
Jul 27 15:51:34.813766 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.813749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5m5f\" (UniqueName: \"kubernetes.io/projected/ea31c5ca-0c82-4d56-b52a-011f3de07eaa-kube-api-access-k5m5f\") pod \"volume-data-source-validator-549c6d54cf-d65gl\" (UID: \"ea31c5ca-0c82-4d56-b52a-011f3de07eaa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"
Jul 27 15:51:34.900096 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.900050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.900250 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.900183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.900250 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:34.900184    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:34.900374 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:34.900302    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:35.400280551 +0000 UTC m=+191.314337100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:34.900735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.900716    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:34.903227 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.903209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"
Jul 27 15:51:34.911051 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.911029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:34.914738 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.914719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"
Jul 27 15:51:34.937652 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.937523    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-h7n4w"]
Jul 27 15:51:34.941467 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:34.941434    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4f5c35a_9ee3_4a33_a2f9_cb3fb9ae79fa.slice/crio-27b06e52b7d09394d76a8e757cdd6d370f102cc32faa993a8f6f137d77a06397 WatchSource:0}: Error finding container 27b06e52b7d09394d76a8e757cdd6d370f102cc32faa993a8f6f137d77a06397: Status 404 returned error can't find the container with id 27b06e52b7d09394d76a8e757cdd6d370f102cc32faa993a8f6f137d77a06397
Jul 27 15:51:34.962667 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:34.962636    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"
Jul 27 15:51:35.050375 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.050303    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qq9gn"]
Jul 27 15:51:35.054332 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:35.054299    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e29607_f7f5_4282_98d9_b2dd928e34d5.slice/crio-8435d67636865e91620ce490cef1eb076d0fa6a0c6a25351ec1ad083ea0e661b WatchSource:0}: Error finding container 8435d67636865e91620ce490cef1eb076d0fa6a0c6a25351ec1ad083ea0e661b: Status 404 returned error can't find the container with id 8435d67636865e91620ce490cef1eb076d0fa6a0c6a25351ec1ad083ea0e661b
Jul 27 15:51:35.123807 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.123777    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks"]
Jul 27 15:51:35.127238 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:35.127210    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod557b7861_bc40_4685_9a4c_1c031bf6d492.slice/crio-02caedbfd01a5523fcf7dbc657505fb25ac9b7b10dc167ad242cda4a00530605 WatchSource:0}: Error finding container 02caedbfd01a5523fcf7dbc657505fb25ac9b7b10dc167ad242cda4a00530605: Status 404 returned error can't find the container with id 02caedbfd01a5523fcf7dbc657505fb25ac9b7b10dc167ad242cda4a00530605
Jul 27 15:51:35.194680 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.194654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h7n4w" event={"ID":"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa","Type":"ContainerStarted","Data":"27b06e52b7d09394d76a8e757cdd6d370f102cc32faa993a8f6f137d77a06397"}
Jul 27 15:51:35.195714 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.195684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks" event={"ID":"557b7861-bc40-4685-9a4c-1c031bf6d492","Type":"ContainerStarted","Data":"02caedbfd01a5523fcf7dbc657505fb25ac9b7b10dc167ad242cda4a00530605"}
Jul 27 15:51:35.196647 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.196624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" event={"ID":"98e29607-f7f5-4282-98d9-b2dd928e34d5","Type":"ContainerStarted","Data":"8435d67636865e91620ce490cef1eb076d0fa6a0c6a25351ec1ad083ea0e661b"}
Jul 27 15:51:35.201944 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.201914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:35.202010 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.201961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:35.202094 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:35.202079    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:36.202061153 +0000 UTC m=+192.116117679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:35.202094 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:35.202083    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:35.202183 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:35.202133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:36.202125446 +0000 UTC m=+192.116181972 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:35.277063 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.277030    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj"]
Jul 27 15:51:35.279124 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.279088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl"]
Jul 27 15:51:35.280562 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:35.280541    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edd01d4_7f69_42d5_a9aa_03dea15f08e5.slice/crio-829e9a436d2ba1e0e8c61b8d5a42b8ea1609ca7ade1683dc2b24b5d44f0ed3b0 WatchSource:0}: Error finding container 829e9a436d2ba1e0e8c61b8d5a42b8ea1609ca7ade1683dc2b24b5d44f0ed3b0: Status 404 returned error can't find the container with id 829e9a436d2ba1e0e8c61b8d5a42b8ea1609ca7ade1683dc2b24b5d44f0ed3b0
Jul 27 15:51:35.281316 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:35.281301    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea31c5ca_0c82_4d56_b52a_011f3de07eaa.slice/crio-4685968735fc442239d5535b3c7bb1e17b70882086c97b83c9db13a544b0c114 WatchSource:0}: Error finding container 4685968735fc442239d5535b3c7bb1e17b70882086c97b83c9db13a544b0c114: Status 404 returned error can't find the container with id 4685968735fc442239d5535b3c7bb1e17b70882086c97b83c9db13a544b0c114
Jul 27 15:51:35.404056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:35.404024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:35.404249 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:35.404181    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:35.404249 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:35.404238    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:36.404224538 +0000 UTC m=+192.318281070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:36.201403 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.201365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj" event={"ID":"3edd01d4-7f69-42d5-a9aa-03dea15f08e5","Type":"ContainerStarted","Data":"43200fb70c92bd95adf5f4f629f9489be34c011decb0dfa1b4dca4f1a32638e8"}
Jul 27 15:51:36.201855 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.201427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj" event={"ID":"3edd01d4-7f69-42d5-a9aa-03dea15f08e5","Type":"ContainerStarted","Data":"829e9a436d2ba1e0e8c61b8d5a42b8ea1609ca7ade1683dc2b24b5d44f0ed3b0"}
Jul 27 15:51:36.202540 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.202517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl" event={"ID":"ea31c5ca-0c82-4d56-b52a-011f3de07eaa","Type":"ContainerStarted","Data":"4685968735fc442239d5535b3c7bb1e17b70882086c97b83c9db13a544b0c114"}
Jul 27 15:51:36.212023 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.211995    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:36.212161 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.212047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:36.212237 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:36.212219    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:38.212200107 +0000 UTC m=+194.126256637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:36.212422 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:36.212400    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:36.212488 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:36.212463    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:38.212445628 +0000 UTC m=+194.126502166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:36.217319 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.216903    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-2tgtj" podStartSLOduration=2.216883486 podStartE2EDuration="2.216883486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:51:36.216754401 +0000 UTC m=+192.130810952" watchObservedRunningTime="2026-07-27 15:51:36.216883486 +0000 UTC m=+192.130940034"
Jul 27 15:51:36.415058 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:36.415021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:36.415235 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:36.415200    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:36.415315 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:36.415281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:38.415259853 +0000 UTC m=+194.329316381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:37.210674 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:37.210602    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl" event={"ID":"ea31c5ca-0c82-4d56-b52a-011f3de07eaa","Type":"ContainerStarted","Data":"0a2eff96ba0cf4e2de58c46b95a2cb068af16d09fca13b1d14320dd5d0e3ab15"}
Jul 27 15:51:37.227553 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:37.226646    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-d65gl" podStartSLOduration=2.043795161 podStartE2EDuration="3.226624427s" totalImagesPullingTime="1.182829266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:51:37.226091686 +0000 UTC m=+193.140148235" watchObservedRunningTime="2026-07-27 15:51:37.226624427 +0000 UTC m=+193.140680980"
Jul 27 15:51:38.230908 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:38.230872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:38.231364 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:38.230998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:38.231364 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:38.231020    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:38.231364 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:38.231088    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:42.231072888 +0000 UTC m=+198.145129434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:38.231364 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:38.231145    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:42.231128825 +0000 UTC m=+198.145185352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:38.432448 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:38.432415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:38.432598 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:38.432565    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:38.432638 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:38.432624    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:42.432610189 +0000 UTC m=+198.346666749 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:40.220598 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.220555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h7n4w" event={"ID":"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa","Type":"ContainerStarted","Data":"a392f0299e7ff51e18658dd48064c32d33fe8243f077e66e6d9ec2ab93b8a30b"}
Jul 27 15:51:40.221997 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.221971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks" event={"ID":"557b7861-bc40-4685-9a4c-1c031bf6d492","Type":"ContainerStarted","Data":"3ac50a1a72e026bbf2a0e6e7d355421ee18a3b4bfe490237c2bb704d6e4dd7ee"}
Jul 27 15:51:40.223329 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.223310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/0.log"
Jul 27 15:51:40.223449 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.223342    2576 generic.go:356] "Generic (PLEG): container finished" podID="98e29607-f7f5-4282-98d9-b2dd928e34d5" containerID="9e471e2d6146a14fc17fcf27cf38c7b23e98e0661b19d4470801a2b115a5c6ce" exitCode=255
Jul 27 15:51:40.223449 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.223393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" event={"ID":"98e29607-f7f5-4282-98d9-b2dd928e34d5","Type":"ContainerDied","Data":"9e471e2d6146a14fc17fcf27cf38c7b23e98e0661b19d4470801a2b115a5c6ce"}
Jul 27 15:51:40.223569 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.223558    2576 scope.go:117] "RemoveContainer" containerID="9e471e2d6146a14fc17fcf27cf38c7b23e98e0661b19d4470801a2b115a5c6ce"
Jul 27 15:51:40.245382 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.245343    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-h7n4w" podStartSLOduration=1.743192247 podStartE2EDuration="6.245329534s" totalImagesPullingTime="4.502137287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:51:40.244977293 +0000 UTC m=+196.159033842" watchObservedRunningTime="2026-07-27 15:51:40.245329534 +0000 UTC m=+196.159386086"
Jul 27 15:51:40.282852 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.282806    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-666b7c558b-j4nks" podStartSLOduration=1.937720667 podStartE2EDuration="6.282788628s" totalImagesPullingTime="4.345067961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:51:40.281404535 +0000 UTC m=+196.195461084" watchObservedRunningTime="2026-07-27 15:51:40.282788628 +0000 UTC m=+196.196845177"
Jul 27 15:51:40.650415 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.650389    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9h6"]
Jul 27 15:51:40.652540 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.652512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.654886 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.654860    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 15:51:40.654990 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.654904    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgjlr\""
Jul 27 15:51:40.655407 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.655392    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 15:51:40.659775 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.659758    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9h6"]
Jul 27 15:51:40.751904 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.751845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b67406a-53cf-455e-ae4d-9329dce3d440-crio-socket\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.751904 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.751873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b67406a-53cf-455e-ae4d-9329dce3d440-data-volume\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.752048 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.751910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.752048 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.751967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.752127 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.752049    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48jf\" (UniqueName: \"kubernetes.io/projected/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-api-access-h48jf\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853083 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853245 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48jf\" (UniqueName: \"kubernetes.io/projected/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-api-access-h48jf\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853245 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b67406a-53cf-455e-ae4d-9329dce3d440-crio-socket\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853245 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b67406a-53cf-455e-ae4d-9329dce3d440-data-volume\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853245 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:40.853202    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:40.853426 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:40.853262    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 15:51:40.853426 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853273    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0b67406a-53cf-455e-ae4d-9329dce3d440-crio-socket\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853426 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:40.853281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:41.353260201 +0000 UTC m=+197.267316744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:40.853426 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.853426 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:40.853326    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:41.353310497 +0000 UTC m=+197.267367035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : configmap "kube-rbac-proxy" not found
Jul 27 15:51:40.853637 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.853488    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0b67406a-53cf-455e-ae4d-9329dce3d440-data-volume\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:40.863481 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:40.863458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48jf\" (UniqueName: \"kubernetes.io/projected/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-api-access-h48jf\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:41.230046 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230022    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/1.log"
Jul 27 15:51:41.230481 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230445    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/0.log"
Jul 27 15:51:41.230537 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230486    2576 generic.go:356] "Generic (PLEG): container finished" podID="98e29607-f7f5-4282-98d9-b2dd928e34d5" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e" exitCode=255
Jul 27 15:51:41.230583 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" event={"ID":"98e29607-f7f5-4282-98d9-b2dd928e34d5","Type":"ContainerDied","Data":"d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"}
Jul 27 15:51:41.230639 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230626    2576 scope.go:117] "RemoveContainer" containerID="9e471e2d6146a14fc17fcf27cf38c7b23e98e0661b19d4470801a2b115a5c6ce"
Jul 27 15:51:41.230878 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.230852    2576 scope.go:117] "RemoveContainer" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"
Jul 27 15:51:41.231056 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:41.231037    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:51:41.356939 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.356902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:41.357091 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:41.357005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:41.357091 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:41.357024    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 15:51:41.357091 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:41.357092    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:42.357070526 +0000 UTC m=+198.271127071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : configmap "kube-rbac-proxy" not found
Jul 27 15:51:41.357281 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:41.357091    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:41.357281 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:41.357160    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:42.357145947 +0000 UTC m=+198.271202491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:42.237193 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.237166    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/1.log"
Jul 27 15:51:42.237552 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.237462    2576 scope.go:117] "RemoveContainer" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"
Jul 27 15:51:42.237619 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.237599    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:51:42.265782 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.265753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:42.265937 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.265834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:42.265937 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.265903    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:42.265937 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.265931    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:50.265918296 +0000 UTC m=+206.179974821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:42.266133 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.265960    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:50.265941948 +0000 UTC m=+206.179998490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:42.366168 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.366136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:42.366343 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.366319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:42.366435 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.366323    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:42.366435 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.366425    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 15:51:42.366547 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.366438    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:44.366415855 +0000 UTC m=+200.280472394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:42.366547 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.366475    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:44.366461467 +0000 UTC m=+200.280517992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : configmap "kube-rbac-proxy" not found
Jul 27 15:51:42.467452 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:42.467414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:42.467597 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.467527    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:42.467597 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:42.467590    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:50.467573148 +0000 UTC m=+206.381629688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:43.196079 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.196049    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-zspf8"]
Jul 27 15:51:43.198795 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.198779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.201551 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.201530    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 15:51:43.201672 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.201586    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 15:51:43.202469 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.202454    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 15:51:43.202541 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.202466    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-n9frr\""
Jul 27 15:51:43.202541 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.202474    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 15:51:43.207705 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.207679    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-zspf8"]
Jul 27 15:51:43.228312 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.228294    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdn7c_a1da3b24-14ce-4aa1-96ff-b045734336da/dns-node-resolver/0.log"
Jul 27 15:51:43.272349 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.272325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-key\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.272637 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.272372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-cabundle\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.272637 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.272410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfqn\" (UniqueName: \"kubernetes.io/projected/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-kube-api-access-6lfqn\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.372749 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.372721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-key\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.372853 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.372768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-cabundle\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.372853 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.372797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfqn\" (UniqueName: \"kubernetes.io/projected/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-kube-api-access-6lfqn\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.373442 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.373422    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-cabundle\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.375041 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.375022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-signing-key\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.383261 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.383243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfqn\" (UniqueName: \"kubernetes.io/projected/69c32e6e-2be3-40db-ae7a-4d0ca52f810a-kube-api-access-6lfqn\") pod \"service-ca-658d7b7c78-zspf8\" (UID: \"69c32e6e-2be3-40db-ae7a-4d0ca52f810a\") " pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.509027 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.508976    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-zspf8"
Jul 27 15:51:43.626446 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:43.626419    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-zspf8"]
Jul 27 15:51:43.628565 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:51:43.628538    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c32e6e_2be3_40db_ae7a_4d0ca52f810a.slice/crio-f0eb05af252f383e036d4df94b8a776b09ddd2f425f9d5cebd8bd357d8655792 WatchSource:0}: Error finding container f0eb05af252f383e036d4df94b8a776b09ddd2f425f9d5cebd8bd357d8655792: Status 404 returned error can't find the container with id f0eb05af252f383e036d4df94b8a776b09ddd2f425f9d5cebd8bd357d8655792
Jul 27 15:51:44.243363 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.243324    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-zspf8" event={"ID":"69c32e6e-2be3-40db-ae7a-4d0ca52f810a","Type":"ContainerStarted","Data":"f0eb05af252f383e036d4df94b8a776b09ddd2f425f9d5cebd8bd357d8655792"}
Jul 27 15:51:44.382225 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.382190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:44.382593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.382246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:44.382593 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:44.382314    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 15:51:44.382593 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:44.382368    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:44.382593 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:44.382383    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:48.382366822 +0000 UTC m=+204.296423352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : configmap "kube-rbac-proxy" not found
Jul 27 15:51:44.382593 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:44.382410    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:48.38239782 +0000 UTC m=+204.296454345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:44.429005 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.428983    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sst8w_73411ea9-8cbf-44ed-b28d-b1dba00cd762/node-ca/0.log"
Jul 27 15:51:44.911199 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.911165    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:44.911369 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.911218    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:51:44.911565 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:44.911543    2576 scope.go:117] "RemoveContainer" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"
Jul 27 15:51:44.911733 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:44.911712    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:51:45.028736 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:45.028711    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74bjz_9881139c-b0ec-4a87-8c39-73f5c23964a3/serve-healthcheck-canary/0.log"
Jul 27 15:51:46.253171 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:46.253140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-zspf8" event={"ID":"69c32e6e-2be3-40db-ae7a-4d0ca52f810a","Type":"ContainerStarted","Data":"cc58b5131b428e01913ec0644a7de5c47744e43e730a12f44292406cd6e001e6"}
Jul 27 15:51:46.272394 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:46.272348    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-zspf8" podStartSLOduration=1.1858227239999999 podStartE2EDuration="3.272333487s" totalImagesPullingTime="2.086510763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:51:46.271418208 +0000 UTC m=+202.185474756" watchObservedRunningTime="2026-07-27 15:51:46.272333487 +0000 UTC m=+202.186390063"
Jul 27 15:51:48.415039 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:48.415002    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:48.415516 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:48.415069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:48.415516 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:48.415146    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 15:51:48.415516 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:48.415199    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:48.415516 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:48.415229    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:56.415207416 +0000 UTC m=+212.329263945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : configmap "kube-rbac-proxy" not found
Jul 27 15:51:48.415516 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:48.415254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:51:56.415242187 +0000 UTC m=+212.329298730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:50.328876 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:50.328843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:50.329258 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:50.328896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:51:50.329258 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:50.329016    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:06.328997625 +0000 UTC m=+222.243054173 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : configmap references non-existent config key: service-ca.crt
Jul 27 15:51:50.329258 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:50.329021    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 15:51:50.329258 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:50.329058    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs podName:395f36f7-8a82-499c-b1e8-f698f52f2594 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:06.329047277 +0000 UTC m=+222.243103803 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs") pod "router-default-6848cdb7df-qbb29" (UID: "395f36f7-8a82-499c-b1e8-f698f52f2594") : secret "router-metrics-certs-default" not found
Jul 27 15:51:50.530687 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:50.530655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:51:50.530867 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:50.530804    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 27 15:51:50.530911 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:50.530867    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert podName:3ac1db04-c9e6-40e4-afda-4f80fb4ad792 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:06.530853632 +0000 UTC m=+222.444910157 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-xnnhk" (UID: "3ac1db04-c9e6-40e4-afda-4f80fb4ad792") : secret "networking-console-plugin-cert" not found
Jul 27 15:51:56.482967 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:56.482934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:56.483355 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:56.483126    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:56.483355 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:51:56.483209    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls podName:0b67406a-53cf-455e-ae4d-9329dce3d440 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:12.483186899 +0000 UTC m=+228.397243427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mv9h6" (UID: "0b67406a-53cf-455e-ae4d-9329dce3d440") : secret "insights-runtime-extractor-tls" not found
Jul 27 15:51:56.483355 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:56.483125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:56.483551 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:56.483535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0b67406a-53cf-455e-ae4d-9329dce3d440-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:51:59.647382 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:51:59.647349    2576 scope.go:117] "RemoveContainer" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"
Jul 27 15:52:00.299757 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.299725    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:52:00.300095 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.300080    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/1.log"
Jul 27 15:52:00.300163 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.300126    2576 generic.go:356] "Generic (PLEG): container finished" podID="98e29607-f7f5-4282-98d9-b2dd928e34d5" containerID="33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386" exitCode=255
Jul 27 15:52:00.300163 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.300154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" event={"ID":"98e29607-f7f5-4282-98d9-b2dd928e34d5","Type":"ContainerDied","Data":"33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386"}
Jul 27 15:52:00.300234 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.300221    2576 scope.go:117] "RemoveContainer" containerID="d87a7fc63e2f1ea428efb7a18968e9ce9afbd2e8db742a4167cc9e9802634c8e"
Jul 27 15:52:00.300562 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:00.300530    2576 scope.go:117] "RemoveContainer" containerID="33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386"
Jul 27 15:52:00.300724 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:52:00.300705    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:52:01.304928 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:01.304902    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:52:02.665223 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.665194    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64849d8666-t8nvz"]
Jul 27 15:52:02.706445 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.706416    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64849d8666-t8nvz"]
Jul 27 15:52:02.706579 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.706547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.710419 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.710393    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 27 15:52:02.710609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.710597    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 27 15:52:02.712049 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.712028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 27 15:52:02.712166 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.712028    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-twmt9\""
Jul 27 15:52:02.715562 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.715544    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 27 15:52:02.833871 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.833835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-image-registry-private-configuration\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.833889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-trusted-ca\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.833953    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-kube-api-access-mwzwv\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.834006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-bound-sa-token\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.834048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/970c6b75-4ec1-49aa-93cc-8d59b99225a9-ca-trust-extracted\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.834135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-installation-pull-secrets\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.834165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-tls\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.834253 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.834184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-certificates\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935398 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-image-registry-private-configuration\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935398 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-trusted-ca\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-kube-api-access-mwzwv\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-bound-sa-token\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935609 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/970c6b75-4ec1-49aa-93cc-8d59b99225a9-ca-trust-extracted\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935767 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935649    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-installation-pull-secrets\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935767 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-tls\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935767 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-certificates\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.935965 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.935945    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/970c6b75-4ec1-49aa-93cc-8d59b99225a9-ca-trust-extracted\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.936479 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.936457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-trusted-ca\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.936599 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.936581    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-certificates\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.937894 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.937873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-image-registry-private-configuration\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.938306 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.938287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-registry-tls\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.938417 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.938398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/970c6b75-4ec1-49aa-93cc-8d59b99225a9-installation-pull-secrets\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.946113 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.946085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-bound-sa-token\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:02.946377 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:02.946361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwzwv\" (UniqueName: \"kubernetes.io/projected/970c6b75-4ec1-49aa-93cc-8d59b99225a9-kube-api-access-mwzwv\") pod \"image-registry-64849d8666-t8nvz\" (UID: \"970c6b75-4ec1-49aa-93cc-8d59b99225a9\") " pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:03.015908 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.015881    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:03.168561 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.168538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64849d8666-t8nvz"]
Jul 27 15:52:03.170241 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:03.170214    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970c6b75_4ec1_49aa_93cc_8d59b99225a9.slice/crio-74c2a21c70ab36cd7fb366743422623e37708dadf0afd22c42510c32d50e6f52 WatchSource:0}: Error finding container 74c2a21c70ab36cd7fb366743422623e37708dadf0afd22c42510c32d50e6f52: Status 404 returned error can't find the container with id 74c2a21c70ab36cd7fb366743422623e37708dadf0afd22c42510c32d50e6f52
Jul 27 15:52:03.312162 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.312128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" event={"ID":"970c6b75-4ec1-49aa-93cc-8d59b99225a9","Type":"ContainerStarted","Data":"fde35ac25d45111c0b822e39103e4002b05e102a1f01e67fbd1f76411b9fdc93"}
Jul 27 15:52:03.312261 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.312186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" event={"ID":"970c6b75-4ec1-49aa-93cc-8d59b99225a9","Type":"ContainerStarted","Data":"74c2a21c70ab36cd7fb366743422623e37708dadf0afd22c42510c32d50e6f52"}
Jul 27 15:52:03.312261 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.312231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:03.332683 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:03.332634    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podStartSLOduration=1.332617553 podStartE2EDuration="1.332617553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:03.332275107 +0000 UTC m=+219.246331655" watchObservedRunningTime="2026-07-27 15:52:03.332617553 +0000 UTC m=+219.246674103"
Jul 27 15:52:04.911728 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:04.911689    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:52:04.912173 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:04.911748    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:52:04.912173 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:04.912050    2576 scope.go:117] "RemoveContainer" containerID="33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386"
Jul 27 15:52:04.912259 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:52:04.912216    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:52:06.364418 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.364382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:06.365004 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.364509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:06.365226 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.365204    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/395f36f7-8a82-499c-b1e8-f698f52f2594-service-ca-bundle\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:06.366650 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.366631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/395f36f7-8a82-499c-b1e8-f698f52f2594-metrics-certs\") pod \"router-default-6848cdb7df-qbb29\" (UID: \"395f36f7-8a82-499c-b1e8-f698f52f2594\") " pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:06.565608 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.565576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:52:06.567756 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.567738    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3ac1db04-c9e6-40e4-afda-4f80fb4ad792-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-xnnhk\" (UID: \"3ac1db04-c9e6-40e4-afda-4f80fb4ad792\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:52:06.615707 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.615641    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:06.740241 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.740196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6848cdb7df-qbb29"]
Jul 27 15:52:06.742528 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:06.742500    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395f36f7_8a82_499c_b1e8_f698f52f2594.slice/crio-7a974017df8ede7649d2c0ddbad55bb0cc9aae77cafaff09914f2eead174c1f6 WatchSource:0}: Error finding container 7a974017df8ede7649d2c0ddbad55bb0cc9aae77cafaff09914f2eead174c1f6: Status 404 returned error can't find the container with id 7a974017df8ede7649d2c0ddbad55bb0cc9aae77cafaff09914f2eead174c1f6
Jul 27 15:52:06.817722 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.817696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"
Jul 27 15:52:06.937579 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:06.937555    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk"]
Jul 27 15:52:06.939182 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:06.939157    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac1db04_c9e6_40e4_afda_4f80fb4ad792.slice/crio-67bc60f6bfa5dc9ba9a455ed3236607a2716c29182e9ba7bf48751f11d6aa8e6 WatchSource:0}: Error finding container 67bc60f6bfa5dc9ba9a455ed3236607a2716c29182e9ba7bf48751f11d6aa8e6: Status 404 returned error can't find the container with id 67bc60f6bfa5dc9ba9a455ed3236607a2716c29182e9ba7bf48751f11d6aa8e6
Jul 27 15:52:07.324771 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.324734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk" event={"ID":"3ac1db04-c9e6-40e4-afda-4f80fb4ad792","Type":"ContainerStarted","Data":"67bc60f6bfa5dc9ba9a455ed3236607a2716c29182e9ba7bf48751f11d6aa8e6"}
Jul 27 15:52:07.325978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.325953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6848cdb7df-qbb29" event={"ID":"395f36f7-8a82-499c-b1e8-f698f52f2594","Type":"ContainerStarted","Data":"1232cde22f0cd11846c6079c9144f24ef641450d6b9a22b84ca3a9ec3358552b"}
Jul 27 15:52:07.326132 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.325998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6848cdb7df-qbb29" event={"ID":"395f36f7-8a82-499c-b1e8-f698f52f2594","Type":"ContainerStarted","Data":"7a974017df8ede7649d2c0ddbad55bb0cc9aae77cafaff09914f2eead174c1f6"}
Jul 27 15:52:07.348620 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.348553    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6848cdb7df-qbb29" podStartSLOduration=33.348534722 podStartE2EDuration="33.348534722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:07.347796823 +0000 UTC m=+223.261853371" watchObservedRunningTime="2026-07-27 15:52:07.348534722 +0000 UTC m=+223.262591271"
Jul 27 15:52:07.616345 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.616250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:07.619094 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:07.619072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:08.330241 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:08.330200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk" event={"ID":"3ac1db04-c9e6-40e4-afda-4f80fb4ad792","Type":"ContainerStarted","Data":"25e08594e599b38434f0af3a3e2092dafeaf03a6886f79eeb04dd90228dd23a0"}
Jul 27 15:52:08.330425 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:08.330407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:08.331502 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:08.331478    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6848cdb7df-qbb29"
Jul 27 15:52:08.347645 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:08.347573    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-xnnhk" podStartSLOduration=33.19665265 podStartE2EDuration="34.347560442s" totalImagesPullingTime="1.150907792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:08.346509733 +0000 UTC m=+224.260566281" watchObservedRunningTime="2026-07-27 15:52:08.347560442 +0000 UTC m=+224.261616989"
Jul 27 15:52:12.508232 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:12.508178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:52:12.510482 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:12.510464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0b67406a-53cf-455e-ae4d-9329dce3d440-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mv9h6\" (UID: \"0b67406a-53cf-455e-ae4d-9329dce3d440\") " pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:52:12.762562 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:12.762466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mv9h6"
Jul 27 15:52:12.888162 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:12.888130    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mv9h6"]
Jul 27 15:52:12.890979 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:12.890937    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b67406a_53cf_455e_ae4d_9329dce3d440.slice/crio-45b5846be1b4e2796678ff3b44a8dbe27ce7b167541a446604aba012ce9f1ac4 WatchSource:0}: Error finding container 45b5846be1b4e2796678ff3b44a8dbe27ce7b167541a446604aba012ce9f1ac4: Status 404 returned error can't find the container with id 45b5846be1b4e2796678ff3b44a8dbe27ce7b167541a446604aba012ce9f1ac4
Jul 27 15:52:13.346845 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:13.346806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9h6" event={"ID":"0b67406a-53cf-455e-ae4d-9329dce3d440","Type":"ContainerStarted","Data":"45b5846be1b4e2796678ff3b44a8dbe27ce7b167541a446604aba012ce9f1ac4"}
Jul 27 15:52:17.646667 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:17.646640    2576 scope.go:117] "RemoveContainer" containerID="33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386"
Jul 27 15:52:17.647139 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:52:17.646820    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-qq9gn_openshift-console-operator(98e29607-f7f5-4282-98d9-b2dd928e34d5)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podUID="98e29607-f7f5-4282-98d9-b2dd928e34d5"
Jul 27 15:52:19.369947 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:19.369904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9h6" event={"ID":"0b67406a-53cf-455e-ae4d-9329dce3d440","Type":"ContainerStarted","Data":"50fc6f675e487c0db80fc8ad650354c7b7ca23cbab94dbb1636c844166a62a92"}
Jul 27 15:52:22.336775 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.335365    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sm5ch"]
Jul 27 15:52:22.348133 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.348038    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.351680 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.351657    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 15:52:22.352214 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.351882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 15:52:22.352214 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.352039    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 15:52:22.353175 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.352933    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t7cjz\""
Jul 27 15:52:22.353175 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.353010    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 15:52:22.353175 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.353056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 15:52:22.353380 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.353210    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 15:52:22.391065 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-textfile\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391065 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-root\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-metrics-client-ca\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-sys\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391471 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-wtmp\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391471 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52c9l\" (UniqueName: \"kubernetes.io/projected/aae884ec-8128-463f-bf0f-dbd4545b3712-kube-api-access-52c9l\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.391471 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.391360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492316 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-sys\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492316 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492325    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-wtmp\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492554 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52c9l\" (UniqueName: \"kubernetes.io/projected/aae884ec-8128-463f-bf0f-dbd4545b3712-kube-api-access-52c9l\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492554 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-sys\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492554 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-wtmp\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492695 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492663    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492750 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-textfile\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492750 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-root\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492854 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492854 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492948 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.492858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-metrics-client-ca\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.492995 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:52:22.492943    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 15:52:22.493045 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:52:22.493034    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls podName:aae884ec-8128-463f-bf0f-dbd4545b3712 nodeName:}" failed. No retries permitted until 2026-07-27 15:52:22.993013138 +0000 UTC m=+238.907069678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls") pod "node-exporter-sm5ch" (UID: "aae884ec-8128-463f-bf0f-dbd4545b3712") : secret "node-exporter-tls" not found
Jul 27 15:52:22.493180 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.493156    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-textfile\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.493273 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.493219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aae884ec-8128-463f-bf0f-dbd4545b3712-root\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.493583 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.493549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-metrics-client-ca\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.493671 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.493609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.495800 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.495778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.505666 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.505636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52c9l\" (UniqueName: \"kubernetes.io/projected/aae884ec-8128-463f-bf0f-dbd4545b3712-kube-api-access-52c9l\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.996824 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.996783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:22.999056 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:22.999027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aae884ec-8128-463f-bf0f-dbd4545b3712-node-exporter-tls\") pod \"node-exporter-sm5ch\" (UID: \"aae884ec-8128-463f-bf0f-dbd4545b3712\") " pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:23.020909 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:23.020879    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:23.021032 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:23.020925    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:23.262150 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:23.262061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm5ch"
Jul 27 15:52:23.274247 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:23.274217    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae884ec_8128_463f_bf0f_dbd4545b3712.slice/crio-b56bb26a18571552e293774372098c7a2b8730a9c9f5a8888d26d5aa78115b2a WatchSource:0}: Error finding container b56bb26a18571552e293774372098c7a2b8730a9c9f5a8888d26d5aa78115b2a: Status 404 returned error can't find the container with id b56bb26a18571552e293774372098c7a2b8730a9c9f5a8888d26d5aa78115b2a
Jul 27 15:52:23.385113 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:23.385074    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm5ch" event={"ID":"aae884ec-8128-463f-bf0f-dbd4545b3712","Type":"ContainerStarted","Data":"b56bb26a18571552e293774372098c7a2b8730a9c9f5a8888d26d5aa78115b2a"}
Jul 27 15:52:23.387289 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:23.387254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9h6" event={"ID":"0b67406a-53cf-455e-ae4d-9329dce3d440","Type":"ContainerStarted","Data":"bbede4e9c7f818b413ceef2e5211cfe5531747ee430bbea9be63cab09096c908"}
Jul 27 15:52:24.321593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:24.321553    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:24.321787 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:24.321619    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:25.397573 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:25.397535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm5ch" event={"ID":"aae884ec-8128-463f-bf0f-dbd4545b3712","Type":"ContainerStarted","Data":"6c2efe9151579700d17a5fbdb1f441739f3a0f79a14f22631a35c64e3960545e"}
Jul 27 15:52:26.402903 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.402867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mv9h6" event={"ID":"0b67406a-53cf-455e-ae4d-9329dce3d440","Type":"ContainerStarted","Data":"47fb061876834639cc6b8e265714abf28a92c23612ca4826fdfa22451c9dbd38"}
Jul 27 15:52:26.404336 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.404315    2576 generic.go:356] "Generic (PLEG): container finished" podID="aae884ec-8128-463f-bf0f-dbd4545b3712" containerID="6c2efe9151579700d17a5fbdb1f441739f3a0f79a14f22631a35c64e3960545e" exitCode=0
Jul 27 15:52:26.404432 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.404342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm5ch" event={"ID":"aae884ec-8128-463f-bf0f-dbd4545b3712","Type":"ContainerDied","Data":"6c2efe9151579700d17a5fbdb1f441739f3a0f79a14f22631a35c64e3960545e"}
Jul 27 15:52:26.424033 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.423987    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mv9h6" podStartSLOduration=33.174568347 podStartE2EDuration="46.423974715s" totalImagesPullingTime="13.249406368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 15:51:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:26.422590171 +0000 UTC m=+242.336646718" watchObservedRunningTime="2026-07-27 15:52:26.423974715 +0000 UTC m=+242.338031266"
Jul 27 15:52:26.615909 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.615875    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5d59984c4f-2znxd"]
Jul 27 15:52:26.618963 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.618944    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.621639 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 15:52:26.621727 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621626    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 15:52:26.621727 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 15:52:26.621727 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621704    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1cn3q6ke0q701\""
Jul 27 15:52:26.621899 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621760    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 15:52:26.621958 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.621912    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5c5fz\""
Jul 27 15:52:26.629602 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.629583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d59984c4f-2znxd"]
Jul 27 15:52:26.728892 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.728825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-metrics-server-audit-profiles\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.728892 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.728865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-tls\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.728892 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.728886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-client-certs\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.729075 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.728913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmgfb\" (UniqueName: \"kubernetes.io/projected/a61d0294-8692-473c-891f-4bcb78776919-kube-api-access-xmgfb\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.729075 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.728969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-client-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.729075 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.729014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a61d0294-8692-473c-891f-4bcb78776919-audit-log\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.729190 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.729081    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830237 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-metrics-server-audit-profiles\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830237 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-tls\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-client-certs\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmgfb\" (UniqueName: \"kubernetes.io/projected/a61d0294-8692-473c-891f-4bcb78776919-kube-api-access-xmgfb\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-client-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a61d0294-8692-473c-891f-4bcb78776919-audit-log\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830405 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830392    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.830781 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.830759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/a61d0294-8692-473c-891f-4bcb78776919-audit-log\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.831060 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.831036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.831285 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.831262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/a61d0294-8692-473c-891f-4bcb78776919-metrics-server-audit-profiles\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.832861 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.832839    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-client-ca-bundle\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.833265 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.833248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-client-certs\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.833471 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.833451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/a61d0294-8692-473c-891f-4bcb78776919-secret-metrics-server-tls\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.839578 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.839551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmgfb\" (UniqueName: \"kubernetes.io/projected/a61d0294-8692-473c-891f-4bcb78776919-kube-api-access-xmgfb\") pod \"metrics-server-5d59984c4f-2znxd\" (UID: \"a61d0294-8692-473c-891f-4bcb78776919\") " pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:26.927367 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:26.927336    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:27.057387 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:27.057350    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5d59984c4f-2znxd"]
Jul 27 15:52:27.060174 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:27.060135    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61d0294_8692_473c_891f_4bcb78776919.slice/crio-737dd1df5670e1b868aa81f78e5b6b60dafe4fcfbc48093713b947c627dfd5b9 WatchSource:0}: Error finding container 737dd1df5670e1b868aa81f78e5b6b60dafe4fcfbc48093713b947c627dfd5b9: Status 404 returned error can't find the container with id 737dd1df5670e1b868aa81f78e5b6b60dafe4fcfbc48093713b947c627dfd5b9
Jul 27 15:52:27.408818 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:27.408732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd" event={"ID":"a61d0294-8692-473c-891f-4bcb78776919","Type":"ContainerStarted","Data":"737dd1df5670e1b868aa81f78e5b6b60dafe4fcfbc48093713b947c627dfd5b9"}
Jul 27 15:52:27.410798 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:27.410773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm5ch" event={"ID":"aae884ec-8128-463f-bf0f-dbd4545b3712","Type":"ContainerStarted","Data":"a664c238708ebc6c1717b1886b48653d304a011197c093786ede98c1a00541f3"}
Jul 27 15:52:27.410906 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:27.410818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm5ch" event={"ID":"aae884ec-8128-463f-bf0f-dbd4545b3712","Type":"ContainerStarted","Data":"94db5e1b11e80b4155f81b64805498c1c91534d64b186157596499e169b6f84f"}
Jul 27 15:52:27.431247 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:27.431203    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sm5ch" podStartSLOduration=4.191360974 podStartE2EDuration="5.431184851s" totalImagesPullingTime="1.169638142s" totalInitContainerRuntime="70.185735ms" isStatefulPod=true podCreationTimestamp="2026-07-27 15:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:27.429873023 +0000 UTC m=+243.343929587" watchObservedRunningTime="2026-07-27 15:52:27.431184851 +0000 UTC m=+243.345241401"
Jul 27 15:52:28.527646 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.527608    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:52:28.531158 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.531135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.537438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.537264    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 15:52:28.537438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.537266    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-69m96\""
Jul 27 15:52:28.537438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.537334    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 15:52:28.537438 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.537345    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 15:52:28.538159 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.537873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 15:52:28.538458 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.538432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 15:52:28.538545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.538439    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 15:52:28.538612 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.538556    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 15:52:28.540448 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.539271    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 15:52:28.540448 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.539297    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 15:52:28.540448 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.540006    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e9i98h8rg4m2n\""
Jul 27 15:52:28.540448 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.540325    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 15:52:28.540680 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.540667    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 15:52:28.541257 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.541087    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 15:52:28.541257 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.541166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 15:52:28.543231 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.543171    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 15:52:28.557752 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.557718    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:52:28.646288 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646387    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646521    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646676 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646676 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646676 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646676 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xqk\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.646847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.647034 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.646869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748116 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748228    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748259 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xqk\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.748558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.748791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749313 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.749289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749615 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.749592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.749988 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.749961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.751543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.751517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.751624 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.751551    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.752013 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.751990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.752172 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.752151    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.752975 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.752942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753079 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.752975    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753197 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.753172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753277 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.753243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753882 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.753857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753965 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.753941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.753965 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.753949    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.754508 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.754481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.755558 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.755537    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.755642 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.755601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.755709 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.755676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.756748 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.756728    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xqk\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk\") pod \"prometheus-k8s-0\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.841807 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.841747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:28.978284 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:28.978253    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:52:28.980488 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:28.980463    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4360a937_a384_4049_9531_cafb50376399.slice/crio-aba7628d7fbe5c4f6c28058db3985c4018a255afd536d3d38ae6bb99e8579081 WatchSource:0}: Error finding container aba7628d7fbe5c4f6c28058db3985c4018a255afd536d3d38ae6bb99e8579081: Status 404 returned error can't find the container with id aba7628d7fbe5c4f6c28058db3985c4018a255afd536d3d38ae6bb99e8579081
Jul 27 15:52:29.419629 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:29.419595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd" event={"ID":"a61d0294-8692-473c-891f-4bcb78776919","Type":"ContainerStarted","Data":"e606a3fbd60af700853cb788172c881ce4c1c4803684b004e717392377b06ca4"}
Jul 27 15:52:29.420895 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:29.420871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"aba7628d7fbe5c4f6c28058db3985c4018a255afd536d3d38ae6bb99e8579081"}
Jul 27 15:52:29.441556 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:29.441514    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd" podStartSLOduration=1.929967246 podStartE2EDuration="3.441496452s" totalImagesPullingTime="1.511529206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:29.440622517 +0000 UTC m=+245.354679066" watchObservedRunningTime="2026-07-27 15:52:29.441496452 +0000 UTC m=+245.355553004"
Jul 27 15:52:30.426662 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:30.426627    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd" exitCode=0
Jul 27 15:52:30.427060 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:30.426713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"}
Jul 27 15:52:32.646608 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:32.646572    2576 scope.go:117] "RemoveContainer" containerID="33ccd787d4c86a5df5bef646934fd2980a17530e54116cad379a0d2ab409b386"
Jul 27 15:52:33.020822 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:33.020789    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:33.021009 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:33.020854    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:34.320319 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.320289    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:34.320667 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.320338    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:34.443978 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.443952    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:52:34.444159 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.444050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" event={"ID":"98e29607-f7f5-4282-98d9-b2dd928e34d5","Type":"ContainerStarted","Data":"2202e5c4aae3c6de36eb919bc76dde438b57bdc36cc8dae323a740bff6fa3f99"}
Jul 27 15:52:34.444422 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.444376    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:52:34.446389 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.446357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"}
Jul 27 15:52:34.446462 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.446412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"}
Jul 27 15:52:34.449342 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.449322    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn"
Jul 27 15:52:34.467418 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:34.467378    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-qq9gn" podStartSLOduration=56.083398304 podStartE2EDuration="1m0.467365129s" totalImagesPullingTime="4.383966825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:34.465710586 +0000 UTC m=+250.379767135" watchObservedRunningTime="2026-07-27 15:52:34.467365129 +0000 UTC m=+250.381421710"
Jul 27 15:52:35.410838 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:35.410797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:52:35.413474 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:35.413448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c549b894-3421-4622-8f97-d756354ab0f8-metrics-certs\") pod \"network-metrics-daemon-2lzpz\" (UID: \"c549b894-3421-4622-8f97-d756354ab0f8\") " pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:52:35.651263 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:35.651226    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x92bc\""
Jul 27 15:52:35.657959 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:35.657933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2lzpz"
Jul 27 15:52:35.803603 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:35.803572    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2lzpz"]
Jul 27 15:52:35.807235 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:52:35.807204    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc549b894_3421_4622_8f97_d756354ab0f8.slice/crio-0ca4794e57cf2878078f03e0d3a1aa60e5befd1dc180d6eb15f11de0776d5ee9 WatchSource:0}: Error finding container 0ca4794e57cf2878078f03e0d3a1aa60e5befd1dc180d6eb15f11de0776d5ee9: Status 404 returned error can't find the container with id 0ca4794e57cf2878078f03e0d3a1aa60e5befd1dc180d6eb15f11de0776d5ee9
Jul 27 15:52:36.459964 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:36.459935    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"}
Jul 27 15:52:36.462504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:36.462373    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2lzpz" event={"ID":"c549b894-3421-4622-8f97-d756354ab0f8","Type":"ContainerStarted","Data":"0ca4794e57cf2878078f03e0d3a1aa60e5befd1dc180d6eb15f11de0776d5ee9"}
Jul 27 15:52:37.468733 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.468690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"}
Jul 27 15:52:37.469301 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.468807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"}
Jul 27 15:52:37.469301 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.468848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerStarted","Data":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"}
Jul 27 15:52:37.470561 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.470536    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2lzpz" event={"ID":"c549b894-3421-4622-8f97-d756354ab0f8","Type":"ContainerStarted","Data":"65cb6d03f7c538b464c7d72680ecfc74cc6b320b8c08e4978aee11682b569b96"}
Jul 27 15:52:37.470682 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.470577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2lzpz" event={"ID":"c549b894-3421-4622-8f97-d756354ab0f8","Type":"ContainerStarted","Data":"de70cf39cdcb88baa7afbc6ffa3fbfc11f33b7739030f8ad9dc61892f1df0827"}
Jul 27 15:52:37.513149 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.513075    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.428453549 podStartE2EDuration="9.513058489s" totalImagesPullingTime="7.062960828s" totalInitContainerRuntime="21.644112ms" isStatefulPod=false podCreationTimestamp="2026-07-27 15:52:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:37.510488107 +0000 UTC m=+253.424544666" watchObservedRunningTime="2026-07-27 15:52:37.513058489 +0000 UTC m=+253.427115038"
Jul 27 15:52:37.539732 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:37.539679    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2lzpz" podStartSLOduration=252.382394921 podStartE2EDuration="4m13.539664284s" totalImagesPullingTime="1.157269363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:52:37.538921184 +0000 UTC m=+253.452977733" watchObservedRunningTime="2026-07-27 15:52:37.539664284 +0000 UTC m=+253.453720832"
Jul 27 15:52:38.842116 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:38.842062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:52:43.021050 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.021002    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:43.021531 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.021074    2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:43.021531 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.021139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:52:43.021649 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.021621    2576 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"fde35ac25d45111c0b822e39103e4002b05e102a1f01e67fbd1f76411b9fdc93"} pod="openshift-image-registry/image-registry-64849d8666-t8nvz" containerMessage="Container registry failed liveness probe, will be restarted"
Jul 27 15:52:43.025093 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.025067    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:43.025212 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:43.025134    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:52:45.505932 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:45.505895    2576 generic.go:356] "Generic (PLEG): container finished" podID="a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa" containerID="a392f0299e7ff51e18658dd48064c32d33fe8243f077e66e6d9ec2ab93b8a30b" exitCode=0
Jul 27 15:52:45.506325 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:45.505971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h7n4w" event={"ID":"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa","Type":"ContainerDied","Data":"a392f0299e7ff51e18658dd48064c32d33fe8243f077e66e6d9ec2ab93b8a30b"}
Jul 27 15:52:45.506367 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:45.506342    2576 scope.go:117] "RemoveContainer" containerID="a392f0299e7ff51e18658dd48064c32d33fe8243f077e66e6d9ec2ab93b8a30b"
Jul 27 15:52:46.511731 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:46.511658    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-h7n4w" event={"ID":"a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa","Type":"ContainerStarted","Data":"222bba1642d1e03bdfe0ff780d8630e1948b83c1dcf6a5779e712170a2f27fca"}
Jul 27 15:52:46.928220 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:46.928141    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:46.928355 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:46.928217    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:52:47.822455 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:47.822427    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdn7c_a1da3b24-14ce-4aa1-96ff-b045734336da/dns-node-resolver/0.log"
Jul 27 15:52:48.426561 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:48.426534    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64849d8666-t8nvz_970c6b75-4ec1-49aa-93cc-8d59b99225a9/registry/0.log"
Jul 27 15:52:49.023376 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:49.023353    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sst8w_73411ea9-8cbf-44ed-b28d-b1dba00cd762/node-ca/0.log"
Jul 27 15:52:49.423455 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:49.423384    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6848cdb7df-qbb29_395f36f7-8a82-499c-b1e8-f698f52f2594/router/0.log"
Jul 27 15:52:49.623242 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:49.623214    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74bjz_9881139c-b0ec-4a87-8c39-73f5c23964a3/serve-healthcheck-canary/0.log"
Jul 27 15:52:53.026117 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:53.026069    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:52:53.026477 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:52:53.026143    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:53:03.026081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:03.026049    2576 patch_prober.go:28] interesting pod/image-registry-64849d8666-t8nvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 27 15:53:03.026460 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:03.026118    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 15:53:04.118757 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:04.118708    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mbxq7" podUID="dc31d968-2a9a-4cb6-8ad5-3b40b9453eac"
Jul 27 15:53:04.573887 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:04.573855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:06.933245 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:06.933215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:53:06.937036 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:06.937009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5d59984c4f-2znxd"
Jul 27 15:53:07.986539 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:07.986503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:07.988802 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:07.988775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dc31d968-2a9a-4cb6-8ad5-3b40b9453eac-metrics-tls\") pod \"dns-default-mbxq7\" (UID: \"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac\") " pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:08.039736 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:08.039700    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" podUID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerName="registry" containerID="cri-o://fde35ac25d45111c0b822e39103e4002b05e102a1f01e67fbd1f76411b9fdc93" gracePeriod=30
Jul 27 15:53:08.177656 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:08.177622    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-748tg\""
Jul 27 15:53:08.185908 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:08.185890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:08.344244 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:08.344221    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mbxq7"]
Jul 27 15:53:08.346377 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:53:08.346354    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc31d968_2a9a_4cb6_8ad5_3b40b9453eac.slice/crio-80cd4864645aed87c7145fb478e14f8ee078691f6283c584d146a69488eb6ebb WatchSource:0}: Error finding container 80cd4864645aed87c7145fb478e14f8ee078691f6283c584d146a69488eb6ebb: Status 404 returned error can't find the container with id 80cd4864645aed87c7145fb478e14f8ee078691f6283c584d146a69488eb6ebb
Jul 27 15:53:08.589517 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:08.589438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxq7" event={"ID":"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac","Type":"ContainerStarted","Data":"80cd4864645aed87c7145fb478e14f8ee078691f6283c584d146a69488eb6ebb"}
Jul 27 15:53:09.594675 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:09.594641    2576 generic.go:356] "Generic (PLEG): container finished" podID="970c6b75-4ec1-49aa-93cc-8d59b99225a9" containerID="fde35ac25d45111c0b822e39103e4002b05e102a1f01e67fbd1f76411b9fdc93" exitCode=0
Jul 27 15:53:09.595026 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:09.594729    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" event={"ID":"970c6b75-4ec1-49aa-93cc-8d59b99225a9","Type":"ContainerDied","Data":"fde35ac25d45111c0b822e39103e4002b05e102a1f01e67fbd1f76411b9fdc93"}
Jul 27 15:53:09.595026 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:09.594775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64849d8666-t8nvz" event={"ID":"970c6b75-4ec1-49aa-93cc-8d59b99225a9","Type":"ContainerStarted","Data":"bdca848dd433e7904b0cc63b0e24171943e54141adc8c7090a12e4e6ae6fdeee"}
Jul 27 15:53:09.595026 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:09.594803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:53:11.604649 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:11.604554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxq7" event={"ID":"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac","Type":"ContainerStarted","Data":"385204527c97e1205a2ef6b45f1628102a58727b8bba4a64bf7ac101d4101f20"}
Jul 27 15:53:11.604649 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:11.604617    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mbxq7" event={"ID":"dc31d968-2a9a-4cb6-8ad5-3b40b9453eac","Type":"ContainerStarted","Data":"4b854bb36bca9f2a312cb6dcf7331273072c4baa837f2ad0042c1a7e4d25d53f"}
Jul 27 15:53:11.605209 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:11.604679    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:11.623716 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:11.623667    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mbxq7" podStartSLOduration=251.738778135 podStartE2EDuration="4m14.623652798s" totalImagesPullingTime="2.884874663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:53:11.622270858 +0000 UTC m=+287.536327420" watchObservedRunningTime="2026-07-27 15:53:11.623652798 +0000 UTC m=+287.537709345"
Jul 27 15:53:21.611318 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:21.611289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mbxq7"
Jul 27 15:53:24.468945 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:24.468912    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:53:24.474386 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:24.474361    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:53:24.480607 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:24.480581    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 15:53:28.842169 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:28.842135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:28.857315 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:28.857290    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:29.684218 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:29.684192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:30.603897 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:30.603865    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64849d8666-t8nvz"
Jul 27 15:53:46.928939 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.928906    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929243    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-web" containerID="cri-o://e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127" gracePeriod=600
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929293    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="thanos-sidecar" containerID="cri-o://71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b" gracePeriod=600
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929299    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="config-reloader" containerID="cri-o://62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734" gracePeriod=600
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929225    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="prometheus" containerID="cri-o://d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4" gracePeriod=600
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929304    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy" containerID="cri-o://e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb" gracePeriod=600
Jul 27 15:53:46.929494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:46.929357    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-thanos" containerID="cri-o://956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695" gracePeriod=600
Jul 27 15:53:47.160245 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.160220    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.220222 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220388 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220233    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220388 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220264    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220501 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220501 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220450    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6xqk\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220501 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220487    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220662 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220607    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220718 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220675    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.220767 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220610    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.221053 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.220982    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.221206 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221085    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221206 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221206 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221220    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221270    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221300    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221331    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221373    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221399    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221450 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221891 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221461    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221891 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221495    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2\") pod \"4360a937-a384-4049-9531-cafb50376399\" (UID: \"4360a937-a384-4049-9531-cafb50376399\") "
Jul 27 15:53:47.221891 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221795    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-metrics-client-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.221891 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.221816    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.222201 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.222069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.222597 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.222374    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.223399 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config" (OuterVolumeSpecName: "config") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.223399 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.223399 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223247    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out" (OuterVolumeSpecName: "config-out") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 15:53:47.223399 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.223668 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.224652 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.223729    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 15:53:47.224652 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.224607    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.224812 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.224742    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.225188 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.225150    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 15:53:47.225467 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.225432    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 15:53:47.225710 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.225687    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk" (OuterVolumeSpecName: "kube-api-access-z6xqk") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "kube-api-access-z6xqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 15:53:47.225844 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.225820    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.226169 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.226147    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.226355 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.226336    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.226776 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.226760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.235470 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.235443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config" (OuterVolumeSpecName: "web-config") pod "4360a937-a384-4049-9531-cafb50376399" (UID: "4360a937-a384-4049-9531-cafb50376399"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 15:53:47.322963 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322938    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.322963 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322960    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322972    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-grpc-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322981    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322990    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.322998    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-tls-assets\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323006    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-web-config\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323014    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323022    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323031    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323039    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323047    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-config-out\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323055    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323063    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-config\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323070    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6xqk\" (UniqueName: \"kubernetes.io/projected/4360a937-a384-4049-9531-cafb50376399-kube-api-access-z6xqk\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323079    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-metrics-client-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323087 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323087    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4360a937-a384-4049-9531-cafb50376399-secret-kube-rbac-proxy\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.323526 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.323095    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4360a937-a384-4049-9531-cafb50376399-prometheus-k8s-db\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743914    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695" exitCode=0
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743941    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb" exitCode=0
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743947    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127" exitCode=0
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743953    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b" exitCode=0
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743958    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734" exitCode=0
Jul 27 15:53:47.743951 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.743963    2576 generic.go:356] "Generic (PLEG): container finished" podID="4360a937-a384-4049-9531-cafb50376399" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4" exitCode=0
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744007    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744002    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744205    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744234    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744286    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"}
Jul 27 15:53:47.744461 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.744310    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4360a937-a384-4049-9531-cafb50376399","Type":"ContainerDied","Data":"aba7628d7fbe5c4f6c28058db3985c4018a255afd536d3d38ae6bb99e8579081"}
Jul 27 15:53:47.752787 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.752769    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.759645 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.759630    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.766143 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.766126    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.771388 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.771365    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:47.772598 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.772583    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.776961 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.776943    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:47.779480 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.779465    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.786146 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.786132    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.792165 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792149    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.792412 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.792393    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.792456 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792419    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.792456 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792437    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.792661 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.792645    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.792704 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792675    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.792704 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792691    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.792880 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.792865    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.792927 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792884    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.792927 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.792897    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.793133 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.793095    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.793173 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793137    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.793173 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793151    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.793341 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.793324    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.793386 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793345    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.793386 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793366    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.793546 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.793532    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.793593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793549    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.793593 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793560    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.793768 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:53:47.793754    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.793811 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793771    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.793811 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793784    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.793968 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793950    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.794010 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.793968    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.794398 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794381    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.794454 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794399    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.794599 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794582    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.794637 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794599    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.794851 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794832    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.794922 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.794852    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.795083 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795067    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.795162 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795084    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.795379 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795360    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.795436 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795379    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.795620 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795601    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.795660 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795622    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.795835 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795817    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.795871 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.795837    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.796065 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796049    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.796119 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796066    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.796315 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796297    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.796360 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796315    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.796511 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796495    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.796546 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796511    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.796697 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796681    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.796740 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796698    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.796892 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796875    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.796930 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.796893    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.797081 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797066    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.797141 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797081    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.797328 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797311    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.797384 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797328    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.797520 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797504    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.797561 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797520    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.797720 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797700    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.797763 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797722    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.797914 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797897    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.797952 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.797914    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.798159 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798143    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.798217 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798159    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.798375 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798354    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.798413 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798375    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.798569 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798550    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.798607 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798571    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.798748 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798732    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.798785 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798748    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.798924 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798908    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.798959 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.798924    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.799096 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799081    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.799096 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799096    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.799305 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799287    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.799344 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799306    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.799496 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799481    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.799539 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799496    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.799689 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799671    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.799735 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799690    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.799889 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.799935 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.799889    2576 scope.go:117] "RemoveContainer" containerID="956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"
Jul 27 15:53:47.800125 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800094    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695"} err="failed to get container status \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": rpc error: code = NotFound desc = could not find container \"956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695\": container with ID starting with 956ddf1bc852740104770c5da1036bd68f7a3c168ace905adb97b8d9745e0695 not found: ID does not exist"
Jul 27 15:53:47.800165 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800128    2576 scope.go:117] "RemoveContainer" containerID="e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"
Jul 27 15:53:47.800353 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800328    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb"} err="failed to get container status \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": rpc error: code = NotFound desc = could not find container \"e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb\": container with ID starting with e0449b5f6e367fae24d0c5db32b0460f4ad360ea38e88aa25483802a89f1a6cb not found: ID does not exist"
Jul 27 15:53:47.800353 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800346    2576 scope.go:117] "RemoveContainer" containerID="e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"
Jul 27 15:53:47.800557 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800542    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127"} err="failed to get container status \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": rpc error: code = NotFound desc = could not find container \"e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127\": container with ID starting with e20d41331adfa813b37cefade8ea59e24555d2ebc650f2300e15b8b1c8615127 not found: ID does not exist"
Jul 27 15:53:47.800599 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800558    2576 scope.go:117] "RemoveContainer" containerID="71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"
Jul 27 15:53:47.800771 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800755    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b"} err="failed to get container status \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": rpc error: code = NotFound desc = could not find container \"71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b\": container with ID starting with 71ecf1e0358693d153434868e0974770be4f6498e9acd1f8a940223679d1188b not found: ID does not exist"
Jul 27 15:53:47.800817 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800772    2576 scope.go:117] "RemoveContainer" containerID="62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"
Jul 27 15:53:47.800967 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800951    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734"} err="failed to get container status \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": rpc error: code = NotFound desc = could not find container \"62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734\": container with ID starting with 62d362fcfb5f0d70e1652e613d4a5bd3f8b143e1fc54b0e84e56ab0f7cfe8734 not found: ID does not exist"
Jul 27 15:53:47.801014 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.800968    2576 scope.go:117] "RemoveContainer" containerID="d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"
Jul 27 15:53:47.801173 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.801156    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4"} err="failed to get container status \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": rpc error: code = NotFound desc = could not find container \"d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4\": container with ID starting with d2cba25491f456893266836605c694d84754e2764d1e05a80cb0e8b4772e87f4 not found: ID does not exist"
Jul 27 15:53:47.801221 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.801174    2576 scope.go:117] "RemoveContainer" containerID="3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"
Jul 27 15:53:47.801429 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.801403    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd"} err="failed to get container status \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": rpc error: code = NotFound desc = could not find container \"3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd\": container with ID starting with 3a10cd1062438d8f895acf0349469f69ca99931f04ac73f051cee44e551109bd not found: ID does not exist"
Jul 27 15:53:47.803580 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803561    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:47.803807 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803795    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="thanos-sidecar"
Jul 27 15:53:47.803847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803809    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="thanos-sidecar"
Jul 27 15:53:47.803847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803824    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="prometheus"
Jul 27 15:53:47.803847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803829    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="prometheus"
Jul 27 15:53:47.803847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803836    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-web"
Jul 27 15:53:47.803847 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803842    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-web"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803848    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803854    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803860    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="init-config-reloader"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803865    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="init-config-reloader"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803872    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-thanos"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803877    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-thanos"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803886    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="config-reloader"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="config-reloader"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803933    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="config-reloader"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803939    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-thanos"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803946    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="prometheus"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803951    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy-web"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803957    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="thanos-sidecar"
Jul 27 15:53:47.803984 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.803964    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4360a937-a384-4049-9531-cafb50376399" containerName="kube-rbac-proxy"
Jul 27 15:53:47.809091 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.809076    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.812208 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812189    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 15:53:47.812460 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812444    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 15:53:47.812530 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812479    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 15:53:47.812530 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812512    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-69m96\""
Jul 27 15:53:47.812530 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812522    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 15:53:47.812668 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812526    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 15:53:47.812668 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812560    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 15:53:47.812777 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812761    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 15:53:47.812844 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812829    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 15:53:47.812898 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812768    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 15:53:47.812959 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.812945    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 15:53:47.813069 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.813055    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e9i98h8rg4m2n\""
Jul 27 15:53:47.813133 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.813120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 15:53:47.813787 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.813770    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 15:53:47.816666 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.816648    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 15:53:47.818434 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.818413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 15:53:47.824182 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.824139    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:47.827886 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.827867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.827980 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.827898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.827980 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.827925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.827980 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.827954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.827984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828136 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828303 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828303 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828185    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828303 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828303 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828233    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828303 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.828513 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.828494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcrxd\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-kube-api-access-jcrxd\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929313 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929313 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcrxd\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-kube-api-access-jcrxd\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.929832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.929940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.930160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.930538 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.930507    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.931018 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.930995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.931078 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.930998    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.931078 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.931027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.932092 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.931641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.932408 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.932375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-config-out\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.932770 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.932744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.932861 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.932826    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.933323 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.933302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.933882 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.933810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.934178 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.934147    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.934283 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.934255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.934497 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.934475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.934565 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.934509    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-web-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.934622 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.934556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/eef6e47d-7d51-4068-bbde-1f558a5612f6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.935407 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.935389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-config\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.935474 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.935407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/eef6e47d-7d51-4068-bbde-1f558a5612f6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.935673 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.935659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:47.942483 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:47.942463    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcrxd\" (UniqueName: \"kubernetes.io/projected/eef6e47d-7d51-4068-bbde-1f558a5612f6-kube-api-access-jcrxd\") pod \"prometheus-k8s-0\" (UID: \"eef6e47d-7d51-4068-bbde-1f558a5612f6\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:48.119281 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.119185    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:53:48.258294 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.258177    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 15:53:48.260889 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:53:48.260862    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef6e47d_7d51_4068_bbde_1f558a5612f6.slice/crio-492f7aeb2e08586cc560aaa03041395645c2e714ff5095827f938d7c03b0ac33 WatchSource:0}: Error finding container 492f7aeb2e08586cc560aaa03041395645c2e714ff5095827f938d7c03b0ac33: Status 404 returned error can't find the container with id 492f7aeb2e08586cc560aaa03041395645c2e714ff5095827f938d7c03b0ac33
Jul 27 15:53:48.651706 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.651664    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4360a937-a384-4049-9531-cafb50376399" path="/var/lib/kubelet/pods/4360a937-a384-4049-9531-cafb50376399/volumes"
Jul 27 15:53:48.749343 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.749312    2576 generic.go:356] "Generic (PLEG): container finished" podID="eef6e47d-7d51-4068-bbde-1f558a5612f6" containerID="a062877ecdaae886ea9b117e11e3870ab1937d9d55a15006b0d9899272528e80" exitCode=0
Jul 27 15:53:48.749530 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.749407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerDied","Data":"a062877ecdaae886ea9b117e11e3870ab1937d9d55a15006b0d9899272528e80"}
Jul 27 15:53:48.749530 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:48.749473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"492f7aeb2e08586cc560aaa03041395645c2e714ff5095827f938d7c03b0ac33"}
Jul 27 15:53:49.756203 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"654550e913ce5465203eca7d9cc4389be9d0b9a96ddb577f5b69d270ebe8e489"}
Jul 27 15:53:49.756587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"c08c3bf06eb41cd1a1b1e465e9993c06cf9a7a9b03f809d7d34b9777e5268f0c"}
Jul 27 15:53:49.756587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"434bb1bc90afe88db22c63dfe25972a2e7d18f133066136b6eb25e84d8d2d242"}
Jul 27 15:53:49.756587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"44f7c020a09039ee802a49d840eaa4a52d76daebfbb081f50550d4c064162501"}
Jul 27 15:53:49.756587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"3f5b9dcaf39aa20c110064b93caa746401223f47545c0ac38f8236393492b7f6"}
Jul 27 15:53:49.756587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.756335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"eef6e47d-7d51-4068-bbde-1f558a5612f6","Type":"ContainerStarted","Data":"73cae1db7eb4b42b820a5c350dcc5fde9e5bdadd445594fb3ef00ba9d5048188"}
Jul 27 15:53:49.791295 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:49.791248    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.770623165 podStartE2EDuration="2.791232769s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.609604ms" isStatefulPod=false podCreationTimestamp="2026-07-27 15:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:53:49.78960359 +0000 UTC m=+325.703660137" watchObservedRunningTime="2026-07-27 15:53:49.791232769 +0000 UTC m=+325.705289333"
Jul 27 15:53:53.120094 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:53:53.120059    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:54:48.120381 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:54:48.120341    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:54:48.135522 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:54:48.135492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:54:49.001305 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:54:49.001280    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 15:57:18.745328 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.745248    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"]
Jul 27 15:57:18.748453 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.748437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.751011 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.750984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 15:57:18.751011 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.750994    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 15:57:18.751216 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.751096    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 15:57:18.751216 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.751124    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2p7wq\""
Jul 27 15:57:18.751216 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.751166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 15:57:18.763361 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.763339    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"]
Jul 27 15:57:18.787403 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.787378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-manifests\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.787493 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.787403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.787493 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.787439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nm8w\" (UniqueName: \"kubernetes.io/projected/07678090-99f6-49cc-8400-9e55b8a6ac4d-kube-api-access-4nm8w\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.787493 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.787458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-tmp\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.787590 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.787525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888137 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nm8w\" (UniqueName: \"kubernetes.io/projected/07678090-99f6-49cc-8400-9e55b8a6ac4d-kube-api-access-4nm8w\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888270 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-tmp\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888270 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888270 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-manifests\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888431 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.888631 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.888602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-manifests\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.890543 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.890515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07678090-99f6-49cc-8400-9e55b8a6ac4d-tmp\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.890696 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.890678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-webhook-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.890763 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.890747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07678090-99f6-49cc-8400-9e55b8a6ac4d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:18.896994 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:18.896973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nm8w\" (UniqueName: \"kubernetes.io/projected/07678090-99f6-49cc-8400-9e55b8a6ac4d-kube-api-access-4nm8w\") pod \"opendatahub-operator-controller-manager-7fc7975cbd-89j4t\" (UID: \"07678090-99f6-49cc-8400-9e55b8a6ac4d\") " pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:19.058017 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:19.057936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:19.187869 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:19.187840    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"]
Jul 27 15:57:19.190907 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:57:19.190876    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07678090_99f6_49cc_8400_9e55b8a6ac4d.slice/crio-6db83f721d5cb5cfb1b1a595c257c9d556f414476644223ce4916e8bbee7e82f WatchSource:0}: Error finding container 6db83f721d5cb5cfb1b1a595c257c9d556f414476644223ce4916e8bbee7e82f: Status 404 returned error can't find the container with id 6db83f721d5cb5cfb1b1a595c257c9d556f414476644223ce4916e8bbee7e82f
Jul 27 15:57:19.192795 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:19.192771    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 15:57:19.531610 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:19.531579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t" event={"ID":"07678090-99f6-49cc-8400-9e55b8a6ac4d","Type":"ContainerStarted","Data":"6db83f721d5cb5cfb1b1a595c257c9d556f414476644223ce4916e8bbee7e82f"}
Jul 27 15:57:22.546055 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:22.546024    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t" event={"ID":"07678090-99f6-49cc-8400-9e55b8a6ac4d","Type":"ContainerStarted","Data":"0ab74ecde2c2d916d525d8c6857936cdc76659b9a9d0482a0aec60bf8cd579f9"}
Jul 27 15:57:23.551949 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:23.551918    2576 generic.go:356] "Generic (PLEG): container finished" podID="07678090-99f6-49cc-8400-9e55b8a6ac4d" containerID="0ab74ecde2c2d916d525d8c6857936cdc76659b9a9d0482a0aec60bf8cd579f9" exitCode=0
Jul 27 15:57:23.552396 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:23.551960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t" event={"ID":"07678090-99f6-49cc-8400-9e55b8a6ac4d","Type":"ContainerDied","Data":"0ab74ecde2c2d916d525d8c6857936cdc76659b9a9d0482a0aec60bf8cd579f9"}
Jul 27 15:57:25.561577 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:25.561538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t" event={"ID":"07678090-99f6-49cc-8400-9e55b8a6ac4d","Type":"ContainerStarted","Data":"a6dd2ff1081ee36b590c0c3152945a6dee82550fce1abfcdc9d8c640f5774f3b"}
Jul 27 15:57:25.561986 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:25.561635    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:25.583265 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:25.583220    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t" podStartSLOduration=2.432458561 podStartE2EDuration="7.583205099s" totalImagesPullingTime="4.073350292s" totalInitContainerRuntime="1.077396246s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:57:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:57:25.580955775 +0000 UTC m=+541.495012323" watchObservedRunningTime="2026-07-27 15:57:25.583205099 +0000 UTC m=+541.497261648"
Jul 27 15:57:36.567650 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:36.567620    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7fc7975cbd-89j4t"
Jul 27 15:57:47.114645 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.114603    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"]
Jul 27 15:57:47.125638 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.125614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.131981 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.131876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.131981 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.131923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-metrics-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.132157 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.131987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjcz\" (UniqueName: \"kubernetes.io/projected/2101e000-95f9-4d62-87de-f08b7b154a82-kube-api-access-4fjcz\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.132157 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.132016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2101e000-95f9-4d62-87de-f08b7b154a82-manager-config\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.132274 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.132187    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 15:57:47.132449 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.132424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 15:57:47.132522 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.132464    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 15:57:47.132522 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.132494    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 15:57:47.133494 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.133473    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 15:57:47.133587 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.133491    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8vvzg\""
Jul 27 15:57:47.200061 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.200032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"]
Jul 27 15:57:47.232999 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.232973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.233171 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.233007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-metrics-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.233171 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.233045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjcz\" (UniqueName: \"kubernetes.io/projected/2101e000-95f9-4d62-87de-f08b7b154a82-kube-api-access-4fjcz\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.233171 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.233062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2101e000-95f9-4d62-87de-f08b7b154a82-manager-config\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.233661 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.233641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2101e000-95f9-4d62-87de-f08b7b154a82-manager-config\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.235633 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.235609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-metrics-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.235714 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.235612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2101e000-95f9-4d62-87de-f08b7b154a82-cert\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.244138 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.244094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjcz\" (UniqueName: \"kubernetes.io/projected/2101e000-95f9-4d62-87de-f08b7b154a82-kube-api-access-4fjcz\") pod \"lws-controller-manager-7c7d956869-nszgh\" (UID: \"2101e000-95f9-4d62-87de-f08b7b154a82\") " pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.434877 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.434794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:47.608183 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.608140    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"]
Jul 27 15:57:47.613839 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:57:47.613812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2101e000_95f9_4d62_87de_f08b7b154a82.slice/crio-ada874f99d35ce3e5fdcfe187c465c7065e9a9876de4cf3acb6f477f291c5590 WatchSource:0}: Error finding container ada874f99d35ce3e5fdcfe187c465c7065e9a9876de4cf3acb6f477f291c5590: Status 404 returned error can't find the container with id ada874f99d35ce3e5fdcfe187c465c7065e9a9876de4cf3acb6f477f291c5590
Jul 27 15:57:47.649497 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:47.649461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh" event={"ID":"2101e000-95f9-4d62-87de-f08b7b154a82","Type":"ContainerStarted","Data":"ada874f99d35ce3e5fdcfe187c465c7065e9a9876de4cf3acb6f477f291c5590"}
Jul 27 15:57:50.455541 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.455511    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"]
Jul 27 15:57:50.464891 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.464868    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.471902 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.471879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6ljc5\""
Jul 27 15:57:50.472029 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.471886    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 27 15:57:50.476191 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.476168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"]
Jul 27 15:57:50.561947 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.561922    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19751475-dae3-4ddd-acec-aec1ec215ed8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.562088 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.561988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2g5l\" (UniqueName: \"kubernetes.io/projected/19751475-dae3-4ddd-acec-aec1ec215ed8-kube-api-access-h2g5l\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.662805 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.662776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2g5l\" (UniqueName: \"kubernetes.io/projected/19751475-dae3-4ddd-acec-aec1ec215ed8-kube-api-access-h2g5l\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.662996 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.662836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19751475-dae3-4ddd-acec-aec1ec215ed8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.665623 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.665589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh" event={"ID":"2101e000-95f9-4d62-87de-f08b7b154a82","Type":"ContainerStarted","Data":"420fc16147bcc5541efdd710ef9826f47637dbf549038911ddac38994d432101"}
Jul 27 15:57:50.665751 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.665663    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:57:50.666184 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.666161    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19751475-dae3-4ddd-acec-aec1ec215ed8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.679591 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.679560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2g5l\" (UniqueName: \"kubernetes.io/projected/19751475-dae3-4ddd-acec-aec1ec215ed8-kube-api-access-h2g5l\") pod \"llmisvc-controller-manager-777c6bd7d5-z88l8\" (UID: \"19751475-dae3-4ddd-acec-aec1ec215ed8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.694900 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.694861    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh" podStartSLOduration=1.216115579 podStartE2EDuration="3.694847525s" totalImagesPullingTime="2.478731946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:57:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:57:50.693875623 +0000 UTC m=+566.607932170" watchObservedRunningTime="2026-07-27 15:57:50.694847525 +0000 UTC m=+566.608904072"
Jul 27 15:57:50.777490 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.777430    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:50.904354 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:50.904327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"]
Jul 27 15:57:50.906667 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:57:50.906636    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod19751475_dae3_4ddd_acec_aec1ec215ed8.slice/crio-c2104ddf1562780c89a99be09f6dc23c5ff63df395fdb49cab7ea334a3473fbf WatchSource:0}: Error finding container c2104ddf1562780c89a99be09f6dc23c5ff63df395fdb49cab7ea334a3473fbf: Status 404 returned error can't find the container with id c2104ddf1562780c89a99be09f6dc23c5ff63df395fdb49cab7ea334a3473fbf
Jul 27 15:57:51.671902 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:51.671864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8" event={"ID":"19751475-dae3-4ddd-acec-aec1ec215ed8","Type":"ContainerStarted","Data":"c2104ddf1562780c89a99be09f6dc23c5ff63df395fdb49cab7ea334a3473fbf"}
Jul 27 15:57:54.684013 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:54.683978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8" event={"ID":"19751475-dae3-4ddd-acec-aec1ec215ed8","Type":"ContainerStarted","Data":"cfbd67fea36fa2c60836f74d79dd6344c231e9e82e3f5f7ba14299afea13ff20"}
Jul 27 15:57:54.684404 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:54.684050    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:57:54.736815 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:57:54.736760    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8" podStartSLOduration=1.903713973 podStartE2EDuration="4.736742556s" totalImagesPullingTime="2.833028583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:57:54.735401076 +0000 UTC m=+570.649457642" watchObservedRunningTime="2026-07-27 15:57:54.736742556 +0000 UTC m=+570.650799104"
Jul 27 15:58:01.675226 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:01.675191    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c7d956869-nszgh"
Jul 27 15:58:24.499650 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:24.499624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:58:24.500187 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:24.499930    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 15:58:25.690348 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:25.690319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z88l8"
Jul 27 15:58:40.668345 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.668310    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:40.670380 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.670365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:40.673143 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.673121    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 15:58:40.673255 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.673126    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-flctt\""
Jul 27 15:58:40.673964 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.673943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 15:58:40.684024 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.684001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:40.688508 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.688484    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qwn\" (UniqueName: \"kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn\") pod \"kuadrant-operator-catalog-dq455\" (UID: \"b07d95a0-c0d3-42d8-95e3-c2586ade672f\") " pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:40.789763 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.789734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qwn\" (UniqueName: \"kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn\") pod \"kuadrant-operator-catalog-dq455\" (UID: \"b07d95a0-c0d3-42d8-95e3-c2586ade672f\") " pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:40.799184 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.799154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qwn\" (UniqueName: \"kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn\") pod \"kuadrant-operator-catalog-dq455\" (UID: \"b07d95a0-c0d3-42d8-95e3-c2586ade672f\") " pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:40.981246 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:40.981211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:41.017767 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.017730    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:41.120503 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.120473    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:41.121292 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:58:41.121260    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07d95a0_c0d3_42d8_95e3_c2586ade672f.slice/crio-51b89c7c3dca51d74e566b44fdb9050146823c61a50506e3b9f1ddc16da02f9d WatchSource:0}: Error finding container 51b89c7c3dca51d74e566b44fdb9050146823c61a50506e3b9f1ddc16da02f9d: Status 404 returned error can't find the container with id 51b89c7c3dca51d74e566b44fdb9050146823c61a50506e3b9f1ddc16da02f9d
Jul 27 15:58:41.229149 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.229116    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sjb6h"]
Jul 27 15:58:41.231983 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.231938    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:41.240650 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.240626    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sjb6h"]
Jul 27 15:58:41.294035 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.294008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tcg\" (UniqueName: \"kubernetes.io/projected/d7a38a8c-b747-4d09-97a8-ed591718d250-kube-api-access-j8tcg\") pod \"kuadrant-operator-catalog-sjb6h\" (UID: \"d7a38a8c-b747-4d09-97a8-ed591718d250\") " pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:41.394932 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.394891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tcg\" (UniqueName: \"kubernetes.io/projected/d7a38a8c-b747-4d09-97a8-ed591718d250-kube-api-access-j8tcg\") pod \"kuadrant-operator-catalog-sjb6h\" (UID: \"d7a38a8c-b747-4d09-97a8-ed591718d250\") " pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:41.404584 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.404560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tcg\" (UniqueName: \"kubernetes.io/projected/d7a38a8c-b747-4d09-97a8-ed591718d250-kube-api-access-j8tcg\") pod \"kuadrant-operator-catalog-sjb6h\" (UID: \"d7a38a8c-b747-4d09-97a8-ed591718d250\") " pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:41.542491 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.542405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:41.667586 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.667551    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sjb6h"]
Jul 27 15:58:41.670451 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:58:41.670420    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a38a8c_b747_4d09_97a8_ed591718d250.slice/crio-2088ca7e0c63adec89f1fa05e1a518521b2e61e67f79834bfc16518f66b47c23 WatchSource:0}: Error finding container 2088ca7e0c63adec89f1fa05e1a518521b2e61e67f79834bfc16518f66b47c23: Status 404 returned error can't find the container with id 2088ca7e0c63adec89f1fa05e1a518521b2e61e67f79834bfc16518f66b47c23
Jul 27 15:58:41.873505 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.873408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq455" event={"ID":"b07d95a0-c0d3-42d8-95e3-c2586ade672f","Type":"ContainerStarted","Data":"51b89c7c3dca51d74e566b44fdb9050146823c61a50506e3b9f1ddc16da02f9d"}
Jul 27 15:58:41.874426 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:41.874398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h" event={"ID":"d7a38a8c-b747-4d09-97a8-ed591718d250","Type":"ContainerStarted","Data":"2088ca7e0c63adec89f1fa05e1a518521b2e61e67f79834bfc16518f66b47c23"}
Jul 27 15:58:44.891970 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:44.891925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq455" event={"ID":"b07d95a0-c0d3-42d8-95e3-c2586ade672f","Type":"ContainerStarted","Data":"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"}
Jul 27 15:58:44.892545 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:44.891986    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dq455" podUID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" containerName="registry-server" containerID="cri-o://67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904" gracePeriod=2
Jul 27 15:58:44.893427 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:44.893398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h" event={"ID":"d7a38a8c-b747-4d09-97a8-ed591718d250","Type":"ContainerStarted","Data":"03efb376aa35b0579db20329c75eb654f14fcab755fb38e7badd5bf6078f951e"}
Jul 27 15:58:44.908838 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:44.908795    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dq455" podStartSLOduration=1.9422740429999998 podStartE2EDuration="4.908781321s" totalImagesPullingTime="2.966507278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:58:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:58:44.908005251 +0000 UTC m=+620.822061795" watchObservedRunningTime="2026-07-27 15:58:44.908781321 +0000 UTC m=+620.822837869"
Jul 27 15:58:44.926167 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:44.926115    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h" podStartSLOduration=1.5195649439999999 podStartE2EDuration="3.926086228s" totalImagesPullingTime="2.406521284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:58:44.924828089 +0000 UTC m=+620.838884660" watchObservedRunningTime="2026-07-27 15:58:44.926086228 +0000 UTC m=+620.840142776"
Jul 27 15:58:45.151247 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.151196    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:45.230013 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.229989    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7qwn\" (UniqueName: \"kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn\") pod \"b07d95a0-c0d3-42d8-95e3-c2586ade672f\" (UID: \"b07d95a0-c0d3-42d8-95e3-c2586ade672f\") "
Jul 27 15:58:45.232114 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.232074    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn" (OuterVolumeSpecName: "kube-api-access-w7qwn") pod "b07d95a0-c0d3-42d8-95e3-c2586ade672f" (UID: "b07d95a0-c0d3-42d8-95e3-c2586ade672f"). InnerVolumeSpecName "kube-api-access-w7qwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 15:58:45.330540 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.330517    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qwn\" (UniqueName: \"kubernetes.io/projected/b07d95a0-c0d3-42d8-95e3-c2586ade672f-kube-api-access-w7qwn\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:58:45.898723 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.898690    2576 generic.go:356] "Generic (PLEG): container finished" podID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" containerID="67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904" exitCode=0
Jul 27 15:58:45.899129 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.898733    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dq455"
Jul 27 15:58:45.899129 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.898773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq455" event={"ID":"b07d95a0-c0d3-42d8-95e3-c2586ade672f","Type":"ContainerDied","Data":"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"}
Jul 27 15:58:45.899129 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.898818    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dq455" event={"ID":"b07d95a0-c0d3-42d8-95e3-c2586ade672f","Type":"ContainerDied","Data":"51b89c7c3dca51d74e566b44fdb9050146823c61a50506e3b9f1ddc16da02f9d"}
Jul 27 15:58:45.899129 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.898842    2576 scope.go:117] "RemoveContainer" containerID="67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"
Jul 27 15:58:45.907925 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.907868    2576 scope.go:117] "RemoveContainer" containerID="67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"
Jul 27 15:58:45.908163 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:58:45.908145    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904\": container with ID starting with 67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904 not found: ID does not exist" containerID="67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"
Jul 27 15:58:45.908235 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.908174    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904"} err="failed to get container status \"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904\": rpc error: code = NotFound desc = could not find container \"67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904\": container with ID starting with 67036ea0d00fb9625f45f5a70174761fbcfbce5730ccba22a23265396b2c2904 not found: ID does not exist"
Jul 27 15:58:45.920680 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.920660    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:45.923822 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:45.923801    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dq455"]
Jul 27 15:58:46.657221 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:46.657170    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" path="/var/lib/kubelet/pods/b07d95a0-c0d3-42d8-95e3-c2586ade672f/volumes"
Jul 27 15:58:51.543343 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:51.543307    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:51.543839 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:51.543379    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:51.564455 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:51.564430    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:58:51.944537 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:58:51.944508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sjb6h"
Jul 27 15:59:13.861487 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.861444    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"]
Jul 27 15:59:13.861997 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.861881    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" containerName="registry-server"
Jul 27 15:59:13.861997 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.861900    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" containerName="registry-server"
Jul 27 15:59:13.861997 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.861987    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b07d95a0-c0d3-42d8-95e3-c2586ade672f" containerName="registry-server"
Jul 27 15:59:13.865130 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.865091    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:13.868006 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.867989    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gb295\""
Jul 27 15:59:13.876920 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.876895    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"]
Jul 27 15:59:13.978561 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.978536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqp5\" (UniqueName: \"kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:13.978680 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:13.978588    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.079998 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.079965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqp5\" (UniqueName: \"kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.080181 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.080032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.080386 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.080367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.093595 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.093570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqp5\" (UniqueName: \"kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.175467 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.175405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:14.309047 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:14.309017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"]
Jul 27 15:59:14.312574 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:59:14.312538    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d0f3ce_877e_4071_9796_dbe3174b9348.slice/crio-1361383d82c6dba050aa32af23af1cc7970c47a8b089ade82e5851a9de2130b6 WatchSource:0}: Error finding container 1361383d82c6dba050aa32af23af1cc7970c47a8b089ade82e5851a9de2130b6: Status 404 returned error can't find the container with id 1361383d82c6dba050aa32af23af1cc7970c47a8b089ade82e5851a9de2130b6
Jul 27 15:59:15.019995 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:15.019958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq" event={"ID":"38d0f3ce-877e-4071-9796-dbe3174b9348","Type":"ContainerStarted","Data":"1361383d82c6dba050aa32af23af1cc7970c47a8b089ade82e5851a9de2130b6"}
Jul 27 15:59:21.047117 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:21.047077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq" event={"ID":"38d0f3ce-877e-4071-9796-dbe3174b9348","Type":"ContainerStarted","Data":"8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633"}
Jul 27 15:59:21.047497 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:21.047198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:21.070111 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:21.070052    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq" podStartSLOduration=1.941039445 podStartE2EDuration="8.070038958s" totalImagesPullingTime="6.128999513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:59:21.068143494 +0000 UTC m=+656.982200043" watchObservedRunningTime="2026-07-27 15:59:21.070038958 +0000 UTC m=+656.984095505"
Jul 27 15:59:22.265417 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.265383    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"]
Jul 27 15:59:22.268732 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.268716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:22.272412 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.272392    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8pg6b\""
Jul 27 15:59:22.284504 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.284482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"]
Jul 27 15:59:22.353354 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.353322    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7bl\" (UniqueName: \"kubernetes.io/projected/d9ad7eb2-85e0-48df-be58-34c511c63589-kube-api-access-pt7bl\") pod \"authorino-operator-75cc8d5cb7-vvpmn\" (UID: \"d9ad7eb2-85e0-48df-be58-34c511c63589\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:22.454352 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.454319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7bl\" (UniqueName: \"kubernetes.io/projected/d9ad7eb2-85e0-48df-be58-34c511c63589-kube-api-access-pt7bl\") pod \"authorino-operator-75cc8d5cb7-vvpmn\" (UID: \"d9ad7eb2-85e0-48df-be58-34c511c63589\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:22.465498 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.465470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7bl\" (UniqueName: \"kubernetes.io/projected/d9ad7eb2-85e0-48df-be58-34c511c63589-kube-api-access-pt7bl\") pod \"authorino-operator-75cc8d5cb7-vvpmn\" (UID: \"d9ad7eb2-85e0-48df-be58-34c511c63589\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:22.578410 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.578326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:22.713947 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:22.713924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"]
Jul 27 15:59:22.716303 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:59:22.716278    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ad7eb2_85e0_48df_be58_34c511c63589.slice/crio-e5df18c51365a5a59fb49429f052f163781fa80ce236a8bff9ae10338736d435 WatchSource:0}: Error finding container e5df18c51365a5a59fb49429f052f163781fa80ce236a8bff9ae10338736d435: Status 404 returned error can't find the container with id e5df18c51365a5a59fb49429f052f163781fa80ce236a8bff9ae10338736d435
Jul 27 15:59:23.056465 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:23.056432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn" event={"ID":"d9ad7eb2-85e0-48df-be58-34c511c63589","Type":"ContainerStarted","Data":"e5df18c51365a5a59fb49429f052f163781fa80ce236a8bff9ae10338736d435"}
Jul 27 15:59:25.067713 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:25.067621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn" event={"ID":"d9ad7eb2-85e0-48df-be58-34c511c63589","Type":"ContainerStarted","Data":"841d4956d9611c05b59dcc7129a757b3617eeed18e089d669a140f4c1035ffb1"}
Jul 27 15:59:25.067713 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:25.067692    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:25.086422 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:25.086374    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn" podStartSLOduration=0.961583582 podStartE2EDuration="3.0863568s" totalImagesPullingTime="2.124773218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:59:25.084240795 +0000 UTC m=+660.998297362" watchObservedRunningTime="2026-07-27 15:59:25.0863568 +0000 UTC m=+661.000413352"
Jul 27 15:59:32.053379 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:32.053350    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:33.772290 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.772256    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"]
Jul 27 15:59:33.772802 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.772420    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" containerName="manager" containerID="cri-o://8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633" gracePeriod=2
Jul 27 15:59:33.783292 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.783265    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"]
Jul 27 15:59:33.789030 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:33.788987    2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"38d0f3ce-877e-4071-9796-dbe3174b9348\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:33.798866 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.798843    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:33.799169 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.799156    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" containerName="manager"
Jul 27 15:59:33.799216 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.799171    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" containerName="manager"
Jul 27 15:59:33.799250 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.799241    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" containerName="manager"
Jul 27 15:59:33.802083 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.802066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.827342 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.827304    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:33.841066 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:33.841029    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" is forbidden: User \"system:node:ip-10-0-135-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-171.ec2.internal' and this object" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:33.854736 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.854715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2cx\" (UniqueName: \"kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.854832 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.854747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.955650 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.955620    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2cx\" (UniqueName: \"kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.955769 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.955658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.955965 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.955950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:33.964828 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:33.964807    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2cx\" (UniqueName: \"kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-x9wsw\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:34.012803 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.012779    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.015053 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:34.015028    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" is forbidden: User \"system:node:ip-10-0-135-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-171.ec2.internal' and this object" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.056470 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.056420    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zqp5\" (UniqueName: \"kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5\") pod \"38d0f3ce-877e-4071-9796-dbe3174b9348\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") "
Jul 27 15:59:34.056555 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.056488    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume\") pod \"38d0f3ce-877e-4071-9796-dbe3174b9348\" (UID: \"38d0f3ce-877e-4071-9796-dbe3174b9348\") "
Jul 27 15:59:34.056840 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.056818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "38d0f3ce-877e-4071-9796-dbe3174b9348" (UID: "38d0f3ce-877e-4071-9796-dbe3174b9348"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 15:59:34.058401 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.058380    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5" (OuterVolumeSpecName: "kube-api-access-6zqp5") pod "38d0f3ce-877e-4071-9796-dbe3174b9348" (UID: "38d0f3ce-877e-4071-9796-dbe3174b9348"). InnerVolumeSpecName "kube-api-access-6zqp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 15:59:34.109358 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.109331    2576 generic.go:356] "Generic (PLEG): container finished" podID="38d0f3ce-877e-4071-9796-dbe3174b9348" containerID="8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633" exitCode=0
Jul 27 15:59:34.109449 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.109378    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.109449 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.109406    2576 scope.go:117] "RemoveContainer" containerID="8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633"
Jul 27 15:59:34.112448 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:34.112413    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" is forbidden: User \"system:node:ip-10-0-135-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-171.ec2.internal' and this object" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.117774 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.117758    2576 scope.go:117] "RemoveContainer" containerID="8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633"
Jul 27 15:59:34.118019 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:34.118000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633\": container with ID starting with 8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633 not found: ID does not exist" containerID="8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633"
Jul 27 15:59:34.118078 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.118026    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633"} err="failed to get container status \"8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633\": rpc error: code = NotFound desc = could not find container \"8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633\": container with ID starting with 8f5584dc351dbd986359c8ab104357b37bb29eeb32ca67d68f0d7bed4295e633 not found: ID does not exist"
Jul 27 15:59:34.120742 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:34.120721    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" is forbidden: User \"system:node:ip-10-0-135-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-171.ec2.internal' and this object" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.154919 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.154897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:34.157247 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.157230    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zqp5\" (UniqueName: \"kubernetes.io/projected/38d0f3ce-877e-4071-9796-dbe3174b9348-kube-api-access-6zqp5\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:59:34.157316 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.157250    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/38d0f3ce-877e-4071-9796-dbe3174b9348-extensions-socket-volume\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:59:34.292061 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.292031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:34.294373 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:59:34.294349    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f69bdbd_04e1_4842_bd26_ac9006050e58.slice/crio-6a57f8778b8617a06022e7c7cfcfa099bfa4352aecbb230f29a6c13dabbcefcc WatchSource:0}: Error finding container 6a57f8778b8617a06022e7c7cfcfa099bfa4352aecbb230f29a6c13dabbcefcc: Status 404 returned error can't find the container with id 6a57f8778b8617a06022e7c7cfcfa099bfa4352aecbb230f29a6c13dabbcefcc
Jul 27 15:59:34.650789 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:34.650708    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jcxjq\" is forbidden: User \"system:node:ip-10-0-135-171.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-171.ec2.internal' and this object" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jcxjq"
Jul 27 15:59:34.652046 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.652024    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38d0f3ce-877e-4071-9796-dbe3174b9348" path="/var/lib/kubelet/pods/38d0f3ce-877e-4071-9796-dbe3174b9348/volumes"
Jul 27 15:59:34.832873 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.832840    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 15:59:34.836797 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.836771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:34.851326 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.851303    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 15:59:34.963688 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.963642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:34.963854 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:34.963734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bn2\" (UniqueName: \"kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.064339 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.064298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bn2\" (UniqueName: \"kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.064506 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.064410    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.064785 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.064764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.076459 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.076432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bn2\" (UniqueName: \"kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2\") pod \"kuadrant-operator-controller-manager-86878c769f-hm986\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.115658 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.115630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" event={"ID":"5f69bdbd-04e1-4842-bd26-ac9006050e58","Type":"ContainerStarted","Data":"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"}
Jul 27 15:59:35.115797 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.115678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" event={"ID":"5f69bdbd-04e1-4842-bd26-ac9006050e58","Type":"ContainerStarted","Data":"6a57f8778b8617a06022e7c7cfcfa099bfa4352aecbb230f29a6c13dabbcefcc"}
Jul 27 15:59:35.115797 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.115731    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:35.134475 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.134419    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" podStartSLOduration=2.134401609 podStartE2EDuration="2.134401609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:59:35.134186601 +0000 UTC m=+671.048243150" watchObservedRunningTime="2026-07-27 15:59:35.134401609 +0000 UTC m=+671.048458159"
Jul 27 15:59:35.147423 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.147399    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:35.277174 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:35.277122    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 15:59:35.280284 ip-10-0-135-171 kubenswrapper[2576]: W0727 15:59:35.280261    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea57d0f_cb42_4add_8c2c_5b0f44277a32.slice/crio-7cbdbd177f52890161dbb6486c219c9efffa4d9852192e79345da00a9bffe347 WatchSource:0}: Error finding container 7cbdbd177f52890161dbb6486c219c9efffa4d9852192e79345da00a9bffe347: Status 404 returned error can't find the container with id 7cbdbd177f52890161dbb6486c219c9efffa4d9852192e79345da00a9bffe347
Jul 27 15:59:36.074818 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:36.074790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vvpmn"
Jul 27 15:59:36.121874 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:36.121837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" event={"ID":"aea57d0f-cb42-4add-8c2c-5b0f44277a32","Type":"ContainerStarted","Data":"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"}
Jul 27 15:59:36.122052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:36.121903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:36.122052 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:36.121929    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" event={"ID":"aea57d0f-cb42-4add-8c2c-5b0f44277a32","Type":"ContainerStarted","Data":"7cbdbd177f52890161dbb6486c219c9efffa4d9852192e79345da00a9bffe347"}
Jul 27 15:59:36.153054 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:36.153007    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" podStartSLOduration=2.152990351 podStartE2EDuration="2.152990351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 15:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 15:59:36.151916839 +0000 UTC m=+672.065973403" watchObservedRunningTime="2026-07-27 15:59:36.152990351 +0000 UTC m=+672.067046898"
Jul 27 15:59:46.124516 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:46.124467    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:47.128392 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.128365    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 15:59:47.199051 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.199005    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:47.199333 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.199253    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" podUID="5f69bdbd-04e1-4842-bd26-ac9006050e58" containerName="manager" containerID="cri-o://3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469" gracePeriod=10
Jul 27 15:59:47.455412 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.455381    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:47.570057 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.569016    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cg2cx\" (UniqueName: \"kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx\") pod \"5f69bdbd-04e1-4842-bd26-ac9006050e58\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") "
Jul 27 15:59:47.570057 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.569139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume\") pod \"5f69bdbd-04e1-4842-bd26-ac9006050e58\" (UID: \"5f69bdbd-04e1-4842-bd26-ac9006050e58\") "
Jul 27 15:59:47.570057 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.569785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5f69bdbd-04e1-4842-bd26-ac9006050e58" (UID: "5f69bdbd-04e1-4842-bd26-ac9006050e58"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 15:59:47.586402 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.586306    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx" (OuterVolumeSpecName: "kube-api-access-cg2cx") pod "5f69bdbd-04e1-4842-bd26-ac9006050e58" (UID: "5f69bdbd-04e1-4842-bd26-ac9006050e58"). InnerVolumeSpecName "kube-api-access-cg2cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 15:59:47.670285 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.670250    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg2cx\" (UniqueName: \"kubernetes.io/projected/5f69bdbd-04e1-4842-bd26-ac9006050e58-kube-api-access-cg2cx\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:59:47.670285 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:47.670276    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5f69bdbd-04e1-4842-bd26-ac9006050e58-extensions-socket-volume\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 15:59:48.178308 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.178272    2576 generic.go:356] "Generic (PLEG): container finished" podID="5f69bdbd-04e1-4842-bd26-ac9006050e58" containerID="3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469" exitCode=0
Jul 27 15:59:48.178694 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.178321    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"
Jul 27 15:59:48.178694 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.178336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" event={"ID":"5f69bdbd-04e1-4842-bd26-ac9006050e58","Type":"ContainerDied","Data":"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"}
Jul 27 15:59:48.178694 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.178384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw" event={"ID":"5f69bdbd-04e1-4842-bd26-ac9006050e58","Type":"ContainerDied","Data":"6a57f8778b8617a06022e7c7cfcfa099bfa4352aecbb230f29a6c13dabbcefcc"}
Jul 27 15:59:48.178694 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.178406    2576 scope.go:117] "RemoveContainer" containerID="3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"
Jul 27 15:59:48.186627 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.186602    2576 scope.go:117] "RemoveContainer" containerID="3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"
Jul 27 15:59:48.186970 ip-10-0-135-171 kubenswrapper[2576]: E0727 15:59:48.186951    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469\": container with ID starting with 3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469 not found: ID does not exist" containerID="3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"
Jul 27 15:59:48.187041 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.186976    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469"} err="failed to get container status \"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469\": rpc error: code = NotFound desc = could not find container \"3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469\": container with ID starting with 3cb7daeef78aeba3b87c43545d089e0ae19075cacbce5ce2a571db469d79d469 not found: ID does not exist"
Jul 27 15:59:48.215890 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.215869    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:48.231066 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.231048    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-x9wsw"]
Jul 27 15:59:48.652152 ip-10-0-135-171 kubenswrapper[2576]: I0727 15:59:48.652092    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f69bdbd-04e1-4842-bd26-ac9006050e58" path="/var/lib/kubelet/pods/5f69bdbd-04e1-4842-bd26-ac9006050e58/volumes"
Jul 27 16:00:06.036597 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.036566    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:00:06.037052 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.036863    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f69bdbd-04e1-4842-bd26-ac9006050e58" containerName="manager"
Jul 27 16:00:06.037052 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.036874    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f69bdbd-04e1-4842-bd26-ac9006050e58" containerName="manager"
Jul 27 16:00:06.037052 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.036941    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f69bdbd-04e1-4842-bd26-ac9006050e58" containerName="manager"
Jul 27 16:00:06.041303 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.041283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.044204 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.044185    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 16:00:06.044390 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.044378    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r6t56\""
Jul 27 16:00:06.058948 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.058927    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:00:06.101637 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.101608    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:00:06.128862 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.128834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lrm\" (UniqueName: \"kubernetes.io/projected/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-kube-api-access-k9lrm\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.128988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.128905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-config-file\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.229237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.229207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-config-file\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.229357 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.229246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lrm\" (UniqueName: \"kubernetes.io/projected/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-kube-api-access-k9lrm\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.229797 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.229778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-config-file\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.241726 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.241703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lrm\" (UniqueName: \"kubernetes.io/projected/e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c-kube-api-access-k9lrm\") pod \"limitador-limitador-54d6d988d5-r2v9s\" (UID: \"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.351204 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.351136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:06.476883 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:06.476855    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:00:06.478879 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:00:06.478856    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c6fb1d_eb12_4cd3_982b_3a36b2982f4c.slice/crio-b74ee730569a91db496fdd6c9a43fb7ccb0d7f4e8ddec3662f4c467d1611b508 WatchSource:0}: Error finding container b74ee730569a91db496fdd6c9a43fb7ccb0d7f4e8ddec3662f4c467d1611b508: Status 404 returned error can't find the container with id b74ee730569a91db496fdd6c9a43fb7ccb0d7f4e8ddec3662f4c467d1611b508
Jul 27 16:00:07.263094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:07.263055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s" event={"ID":"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c","Type":"ContainerStarted","Data":"b74ee730569a91db496fdd6c9a43fb7ccb0d7f4e8ddec3662f4c467d1611b508"}
Jul 27 16:00:11.288706 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:11.288673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s" event={"ID":"e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c","Type":"ContainerStarted","Data":"9bf9aa8d9360d7558d6f1e09971d43a170b8cc4561ac38b1a0d96c55a0d3dc50"}
Jul 27 16:00:11.289097 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:11.288795    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:00:11.308953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:11.308903    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s" podStartSLOduration=1.488298944 podStartE2EDuration="5.30888983s" totalImagesPullingTime="3.820590886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:00:11.307016296 +0000 UTC m=+707.221072848" watchObservedRunningTime="2026-07-27 16:00:11.30888983 +0000 UTC m=+707.222946430"
Jul 27 16:00:22.295071 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:00:22.295001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"
Jul 27 16:01:25.152763 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:01:25.152729    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:01:45.520732 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:01:45.520693    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:01:54.206527 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:01:54.206435    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:02:06.593122 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:06.593070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:02:09.794822 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.794784    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"]
Jul 27 16:02:09.801349 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.801321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.806897 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.806872    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 16:02:09.807253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.807229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9b5nx\""
Jul 27 16:02:09.807450 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.807431    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 16:02:09.807900 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.807874    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 27 16:02:09.809034 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.809012    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"]
Jul 27 16:02:09.866266 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866231    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccljz\" (UniqueName: \"kubernetes.io/projected/36d637e8-fb73-4129-b094-b402be7cb32b-kube-api-access-ccljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.866446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.866446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.866446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.866446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.866446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.866403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d637e8-fb73-4129-b094-b402be7cb32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.967858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.967858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d637e8-fb73-4129-b094-b402be7cb32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccljz\" (UniqueName: \"kubernetes.io/projected/36d637e8-fb73-4129-b094-b402be7cb32b-kube-api-access-ccljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.967989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968434 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.968412    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968475 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.968424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.968557 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.968533    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.970157 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.970135    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/36d637e8-fb73-4129-b094-b402be7cb32b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.970425 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.970408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36d637e8-fb73-4129-b094-b402be7cb32b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:09.977821 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:09.977794    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccljz\" (UniqueName: \"kubernetes.io/projected/36d637e8-fb73-4129-b094-b402be7cb32b-kube-api-access-ccljz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r\" (UID: \"36d637e8-fb73-4129-b094-b402be7cb32b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:10.114242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:10.114146    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:10.247547 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:10.247506    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"]
Jul 27 16:02:10.250792 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:02:10.250764    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d637e8_fb73_4129_b094_b402be7cb32b.slice/crio-726d119c09ca660aa93bef7a663f25fab0da447a2af282fd97ee963ad1a8cbe2 WatchSource:0}: Error finding container 726d119c09ca660aa93bef7a663f25fab0da447a2af282fd97ee963ad1a8cbe2: Status 404 returned error can't find the container with id 726d119c09ca660aa93bef7a663f25fab0da447a2af282fd97ee963ad1a8cbe2
Jul 27 16:02:10.806070 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:10.806029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r" event={"ID":"36d637e8-fb73-4129-b094-b402be7cb32b","Type":"ContainerStarted","Data":"726d119c09ca660aa93bef7a663f25fab0da447a2af282fd97ee963ad1a8cbe2"}
Jul 27 16:02:11.207453 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:11.207417    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:02:11.812000 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:11.811971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r" event={"ID":"36d637e8-fb73-4129-b094-b402be7cb32b","Type":"ContainerStarted","Data":"b34981b2c16cd7d6cd879aa0c788a37f8488216d74e01a5df6bece42009bf8e6"}
Jul 27 16:02:11.830677 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:11.830631    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r" podStartSLOduration=1.7502057720000002 podStartE2EDuration="2.830614407s" totalImagesPullingTime="1.080408635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:02:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:02:11.829635473 +0000 UTC m=+827.743692023" watchObservedRunningTime="2026-07-27 16:02:11.830614407 +0000 UTC m=+827.744670955"
Jul 27 16:02:17.713159 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:17.713124    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:02:20.114670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:20.114630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:20.115040 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:20.114698    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:20.127032 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:20.127011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:20.865526 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:20.865498    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r"
Jul 27 16:02:25.980614 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:25.980583    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"]
Jul 27 16:02:25.983167 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:25.983150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:25.989538 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:25.989473    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 16:02:26.013075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.013053    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"]
Jul 27 16:02:26.108488 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.108627 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108506    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.108627 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.108627 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqvd\" (UniqueName: \"kubernetes.io/projected/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-kube-api-access-9sqvd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.108732 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.108732 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.108663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.209876 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqvd\" (UniqueName: \"kubernetes.io/projected/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-kube-api-access-9sqvd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.209988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210378 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.210356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210447 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.210409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.210500 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.210445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.212088 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.212068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.212397 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.212379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.218973 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.218954    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqvd\" (UniqueName: \"kubernetes.io/projected/4b7c0030-5c11-4f4b-a30e-3497fe25a6b4-kube-api-access-9sqvd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp\" (UID: \"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.292634 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.292580    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:26.427622 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.427593    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"]
Jul 27 16:02:26.430090 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:02:26.430065    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7c0030_5c11_4f4b_a30e_3497fe25a6b4.slice/crio-1843e55e2c8fd9ab37698e23cb24b4c09afd3fe89be50c2c5b9eece8d4433153 WatchSource:0}: Error finding container 1843e55e2c8fd9ab37698e23cb24b4c09afd3fe89be50c2c5b9eece8d4433153: Status 404 returned error can't find the container with id 1843e55e2c8fd9ab37698e23cb24b4c09afd3fe89be50c2c5b9eece8d4433153
Jul 27 16:02:26.431931 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.431914    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:02:26.879743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.879667    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp" event={"ID":"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4","Type":"ContainerStarted","Data":"29176089e005df5d0695abeb42647ea6917a4684203679e947fd347c33e7dc39"}
Jul 27 16:02:26.879743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.879710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp" event={"ID":"4b7c0030-5c11-4f4b-a30e-3497fe25a6b4","Type":"ContainerStarted","Data":"1843e55e2c8fd9ab37698e23cb24b4c09afd3fe89be50c2c5b9eece8d4433153"}
Jul 27 16:02:26.900426 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:26.900379    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp" podStartSLOduration=1.7305012629999998 podStartE2EDuration="1.900362852s" totalImagesPullingTime="169.861589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:02:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:02:26.898498523 +0000 UTC m=+842.812555072" watchObservedRunningTime="2026-07-27 16:02:26.900362852 +0000 UTC m=+842.814419400"
Jul 27 16:02:28.413548 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:28.413515    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:02:36.293207 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:36.293169    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:36.293629 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:36.293229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:36.305921 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:36.305893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:02:36.933804 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:02:36.933771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp"
Jul 27 16:03:24.525160 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:03:24.525064    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:03:24.527545 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:03:24.527524    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:04:21.303910 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:04:21.303872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:04:31.098890 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:04:31.098854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:04:41.403426 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:04:41.403389    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:06:34.903578 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:06:34.903497    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:06:44.205212 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:06:44.205177    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:07:14.603796 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:07:14.603766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:07:23.898092 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:07:23.898057    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:07:32.304122 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:07:32.304071    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:07:41.502340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:07:41.502306    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:07:51.498756 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:07:51.498678    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:08:01.812727 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:01.812689    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:08:13.090186 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:13.090153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:08:22.505237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:22.505203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:08:24.551542 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:24.551513    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:08:24.553847 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:24.553820    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:08:45.308014 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:45.307979    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:08:57.000495 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:08:57.000465    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:09:21.105334 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:09:21.105254    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:09:31.094950 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:09:31.094915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:09:46.306809 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:09:46.306774    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:09:57.996759 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:09:57.996725    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:10:07.305885 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:07.305853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:10:17.590861 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:17.590822    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:10:26.306228 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:26.306194    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:10:51.924766 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:51.924659    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 16:10:51.925357 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:51.924881    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" podUID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" containerName="manager" containerID="cri-o://f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6" gracePeriod=10
Jul 27 16:10:55.781232 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.781206    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 16:10:55.852643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.852574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2bn2\" (UniqueName: \"kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2\") pod \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") "
Jul 27 16:10:55.852643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.852614    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume\") pod \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\" (UID: \"aea57d0f-cb42-4add-8c2c-5b0f44277a32\") "
Jul 27 16:10:55.852982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.852962    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aea57d0f-cb42-4add-8c2c-5b0f44277a32" (UID: "aea57d0f-cb42-4add-8c2c-5b0f44277a32"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:10:55.854571 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.854544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2" (OuterVolumeSpecName: "kube-api-access-n2bn2") pod "aea57d0f-cb42-4add-8c2c-5b0f44277a32" (UID: "aea57d0f-cb42-4add-8c2c-5b0f44277a32"). InnerVolumeSpecName "kube-api-access-n2bn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:10:55.953486 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.953449    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bn2\" (UniqueName: \"kubernetes.io/projected/aea57d0f-cb42-4add-8c2c-5b0f44277a32-kube-api-access-n2bn2\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:10:55.953486 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:55.953484    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aea57d0f-cb42-4add-8c2c-5b0f44277a32-extensions-socket-volume\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:10:56.184276 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.184197    2576 generic.go:356] "Generic (PLEG): container finished" podID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" containerID="f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6" exitCode=0
Jul 27 16:10:56.184276 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.184240    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"
Jul 27 16:10:56.184276 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.184272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" event={"ID":"aea57d0f-cb42-4add-8c2c-5b0f44277a32","Type":"ContainerDied","Data":"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"}
Jul 27 16:10:56.184481 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.184313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986" event={"ID":"aea57d0f-cb42-4add-8c2c-5b0f44277a32","Type":"ContainerDied","Data":"7cbdbd177f52890161dbb6486c219c9efffa4d9852192e79345da00a9bffe347"}
Jul 27 16:10:56.184481 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.184335    2576 scope.go:117] "RemoveContainer" containerID="f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"
Jul 27 16:10:56.194209 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.194190    2576 scope.go:117] "RemoveContainer" containerID="f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"
Jul 27 16:10:56.194457 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:10:56.194436    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6\": container with ID starting with f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6 not found: ID does not exist" containerID="f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"
Jul 27 16:10:56.194522 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.194469    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6"} err="failed to get container status \"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6\": rpc error: code = NotFound desc = could not find container \"f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6\": container with ID starting with f40b3627a6bb84961280ec1cfc5eacab3df3cf4ab0c1971dbf1d56fa952042e6 not found: ID does not exist"
Jul 27 16:10:56.212769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.212719    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 16:10:56.218855 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.218829    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hm986"]
Jul 27 16:10:56.660277 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:10:56.660241    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" path="/var/lib/kubelet/pods/aea57d0f-cb42-4add-8c2c-5b0f44277a32/volumes"
Jul 27 16:11:53.113436 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.113397    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"]
Jul 27 16:11:53.113850 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.113731    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" containerName="manager"
Jul 27 16:11:53.113850 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.113743    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" containerName="manager"
Jul 27 16:11:53.113850 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.113805    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea57d0f-cb42-4add-8c2c-5b0f44277a32" containerName="manager"
Jul 27 16:11:53.116340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.116324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.119694 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.119673    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gb295\""
Jul 27 16:11:53.134011 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.133981    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"]
Jul 27 16:11:53.136518 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.136492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b86168-5e30-4219-93be-d35e6abb553a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.136635 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.136552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fp4\" (UniqueName: \"kubernetes.io/projected/46b86168-5e30-4219-93be-d35e6abb553a-kube-api-access-v2fp4\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.237520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.237492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b86168-5e30-4219-93be-d35e6abb553a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.237678 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.237533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fp4\" (UniqueName: \"kubernetes.io/projected/46b86168-5e30-4219-93be-d35e6abb553a-kube-api-access-v2fp4\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.237905 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.237884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/46b86168-5e30-4219-93be-d35e6abb553a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.248815 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.248795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fp4\" (UniqueName: \"kubernetes.io/projected/46b86168-5e30-4219-93be-d35e6abb553a-kube-api-access-v2fp4\") pod \"kuadrant-operator-controller-manager-86878c769f-qq5p2\" (UID: \"46b86168-5e30-4219-93be-d35e6abb553a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.434566 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.434489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:53.564374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:53.564351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"]
Jul 27 16:11:53.566861 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:11:53.566834    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b86168_5e30_4219_93be_d35e6abb553a.slice/crio-b0af13c637c0b8e35303530a48f61ae6a3cba2ca14e14ff25d39661ccf6e4f6e WatchSource:0}: Error finding container b0af13c637c0b8e35303530a48f61ae6a3cba2ca14e14ff25d39661ccf6e4f6e: Status 404 returned error can't find the container with id b0af13c637c0b8e35303530a48f61ae6a3cba2ca14e14ff25d39661ccf6e4f6e
Jul 27 16:11:54.439747 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:54.439712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2" event={"ID":"46b86168-5e30-4219-93be-d35e6abb553a","Type":"ContainerStarted","Data":"b13af46d9e16d4024bd6f275957a4a536996ffeae12cb7bf890e9b244938e8fa"}
Jul 27 16:11:54.439747 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:54.439764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2" event={"ID":"46b86168-5e30-4219-93be-d35e6abb553a","Type":"ContainerStarted","Data":"b0af13c637c0b8e35303530a48f61ae6a3cba2ca14e14ff25d39661ccf6e4f6e"}
Jul 27 16:11:54.440312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:54.439807    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:11:54.467442 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:11:54.467392    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2" podStartSLOduration=1.467377126 podStartE2EDuration="1.467377126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:11:54.464942566 +0000 UTC m=+1410.378999114" watchObservedRunningTime="2026-07-27 16:11:54.467377126 +0000 UTC m=+1410.381433674"
Jul 27 16:12:05.446269 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:05.446238    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qq5p2"
Jul 27 16:12:14.091845 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:14.091810    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:12:24.122158 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:24.122061    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:12:35.295632 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:35.295594    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:12:45.696257 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:45.696228    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:12:57.099728 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:12:57.099685    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:07.295494 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:07.295458    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:17.493857 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:17.493815    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:24.576847 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:24.576815    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:13:24.580127 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:24.580086    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:13:26.907884 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:26.907848    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:38.592508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:38.592472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:47.128936 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:47.128854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:13:57.895705 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:13:57.895667    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:14:20.201153 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:20.201120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:14:29.395717 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:29.395681    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:14:46.309652 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.309617    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:14:46.313040 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.313018    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.315695 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.315678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-80d812c7-kserve-self-signed-certs\""
Jul 27 16:14:46.334177 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.334156    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:14:46.432718 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.432682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.432868 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.432764    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.432868 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.432848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.432969 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.432885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dck24\" (UniqueName: \"kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.432969 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.432916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.433035 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.433007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533504 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dck24\" (UniqueName: \"kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.533968 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.534175 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.533993    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.534175 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.534038    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.536194 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.536167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.536337 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.536320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.543020 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.542997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dck24\" (UniqueName: \"kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24\") pod \"e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") " pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.621965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.621902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:46.756634 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:46.756610    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:14:46.758435 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:14:46.758405    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbf54732_0297_475c_a5f9_0df7b5e00c5d.slice/crio-9649671319a230b514ecf85513feb72b2c8ae028bbf9eed87c91cadc7c6b86a8 WatchSource:0}: Error finding container 9649671319a230b514ecf85513feb72b2c8ae028bbf9eed87c91cadc7c6b86a8: Status 404 returned error can't find the container with id 9649671319a230b514ecf85513feb72b2c8ae028bbf9eed87c91cadc7c6b86a8
Jul 27 16:14:47.217866 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:47.217817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" event={"ID":"bbf54732-0297-475c-a5f9-0df7b5e00c5d","Type":"ContainerStarted","Data":"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"}
Jul 27 16:14:47.217866 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:47.217871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" event={"ID":"bbf54732-0297-475c-a5f9-0df7b5e00c5d","Type":"ContainerStarted","Data":"9649671319a230b514ecf85513feb72b2c8ae028bbf9eed87c91cadc7c6b86a8"}
Jul 27 16:14:47.241667 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:47.241614    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" podStartSLOduration=1.241600007 podStartE2EDuration="1.241600007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:14:47.239784079 +0000 UTC m=+1583.153840627" watchObservedRunningTime="2026-07-27 16:14:47.241600007 +0000 UTC m=+1583.155656592"
Jul 27 16:14:49.411958 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:49.411925    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:14:56.622802 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:56.622720    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:56.622802 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:56.622768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:56.635240 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:56.635218    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:14:57.276160 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:14:57.276133    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:15:05.738860 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:05.738823    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:15:34.168529 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:34.168455    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:15:37.997957 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:37.997923    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:15:49.187520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.187480    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" podUID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" containerName="main" containerID="cri-o://f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636" gracePeriod=45
Jul 27 16:15:49.444190 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.444164    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:15:49.490959 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.490929    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491115 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491001    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491115 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491115 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dck24\" (UniqueName: \"kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491145    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491186    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home\") pod \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\" (UID: \"bbf54732-0297-475c-a5f9-0df7b5e00c5d\") "
Jul 27 16:15:49.491379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491142    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:15:49.491379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491339    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:15:49.491506 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491488    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:49.491563 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491512    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:49.491563 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.491544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home" (OuterVolumeSpecName: "home") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:15:49.493433 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.493404    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm" (OuterVolumeSpecName: "dshm") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:15:49.493692 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.493670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24" (OuterVolumeSpecName: "kube-api-access-dck24") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "kube-api-access-dck24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:15:49.493766 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.493672    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbf54732-0297-475c-a5f9-0df7b5e00c5d" (UID: "bbf54732-0297-475c-a5f9-0df7b5e00c5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:15:49.511375 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.511353    2576 generic.go:356] "Generic (PLEG): container finished" podID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" containerID="f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636" exitCode=0
Jul 27 16:15:49.511468 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.511397    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"
Jul 27 16:15:49.511468 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.511438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" event={"ID":"bbf54732-0297-475c-a5f9-0df7b5e00c5d","Type":"ContainerDied","Data":"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"}
Jul 27 16:15:49.511532 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.511478    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56" event={"ID":"bbf54732-0297-475c-a5f9-0df7b5e00c5d","Type":"ContainerDied","Data":"9649671319a230b514ecf85513feb72b2c8ae028bbf9eed87c91cadc7c6b86a8"}
Jul 27 16:15:49.511532 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.511497    2576 scope.go:117] "RemoveContainer" containerID="f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"
Jul 27 16:15:49.519843 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.519826    2576 scope.go:117] "RemoveContainer" containerID="f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"
Jul 27 16:15:49.520088 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:15:49.520068    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636\": container with ID starting with f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636 not found: ID does not exist" containerID="f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"
Jul 27 16:15:49.520159 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.520095    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636"} err="failed to get container status \"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636\": rpc error: code = NotFound desc = could not find container \"f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636\": container with ID starting with f329a7a826a82372b47095737e84efd591be38f781101b61b6d6eaf5f0c08636 not found: ID does not exist"
Jul 27 16:15:49.532694 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.532675    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:15:49.537423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.537398    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-80d812c7-kserve-547dc9d575-fsn56"]
Jul 27 16:15:49.592407 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.592380    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dck24\" (UniqueName: \"kubernetes.io/projected/bbf54732-0297-475c-a5f9-0df7b5e00c5d-kube-api-access-dck24\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:49.592407 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.592402    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:49.592544 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.592412    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf54732-0297-475c-a5f9-0df7b5e00c5d-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:49.592544 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:49.592420    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf54732-0297-475c-a5f9-0df7b5e00c5d-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:15:50.652552 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:50.652519    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" path="/var/lib/kubelet/pods/bbf54732-0297-475c-a5f9-0df7b5e00c5d/volumes"
Jul 27 16:15:51.302253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:51.302208    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:15:58.200889 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:15:58.200862    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:13.494443 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:13.494410    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:22.902520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:22.902485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:31.888967 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:31.888927    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:39.799480 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:39.799444    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:49.497697 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:49.497621    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:16:58.700900 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:16:58.700870    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:17:11.899198 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:17:11.899161    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:17:21.114055 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:17:21.114022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:17:30.799302 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:17:30.799268    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:17:43.299611 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:17:43.299573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:17:54.101686 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:17:54.101652    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:02.298535 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:02.298503    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:14.103013 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:14.102978    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:23.798278 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:23.798241    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:24.602997 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:24.602965    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:18:24.610626 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:24.610602    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:18:32.805865 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:32.805833    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:40.796321 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:40.796287    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:18:54.611599 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:18:54.611569    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:19:04.092480 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:04.092442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:19:13.601928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:13.601898    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:19:16.883207 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.883177    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:16.883639 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.883620    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" containerName="main"
Jul 27 16:19:16.883681 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.883643    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" containerName="main"
Jul 27 16:19:16.883743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.883734    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf54732-0297-475c-a5f9-0df7b5e00c5d" containerName="main"
Jul 27 16:19:16.891678 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.891650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:16.895428 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.895399    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:19:16.895562 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.895455    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:19:16.895562 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.895399    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9ab13729-serving-cert\""
Jul 27 16:19:16.895774 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.895758    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:19:16.897120 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.897088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:16.999201 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.999168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:16.999376 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.999240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:16.999376 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:16.999259    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47xcj\" (UniqueName: \"kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.100524 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.100487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.100691 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.100580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.100691 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.100609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47xcj\" (UniqueName: \"kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.100771 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:19:17.100730    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-serving-cert: secret "maas-api-e2e-ait-9ab13729-serving-cert" not found
Jul 27 16:19:17.100820 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:19:17.100810    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls podName:0621648c-28a5-416b-895e-898baf34a3ae nodeName:}" failed. No retries permitted until 2026-07-27 16:19:17.600790243 +0000 UTC m=+1853.514846770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls") pod "maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" (UID: "0621648c-28a5-416b-895e-898baf34a3ae") : secret "maas-api-e2e-ait-9ab13729-serving-cert" not found
Jul 27 16:19:17.101171 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.101154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.114064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.114040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47xcj\" (UniqueName: \"kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.605284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.605247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.607972 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.607944    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") pod \"maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.804796 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.804764    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:17.939393 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.939368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:17.942484 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:19:17.942441    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0621648c_28a5_416b_895e_898baf34a3ae.slice/crio-d046e4f2557ec47a66721b74acdae4d7f08cc25bf17181fbebe37f5685d7b07d WatchSource:0}: Error finding container d046e4f2557ec47a66721b74acdae4d7f08cc25bf17181fbebe37f5685d7b07d: Status 404 returned error can't find the container with id d046e4f2557ec47a66721b74acdae4d7f08cc25bf17181fbebe37f5685d7b07d
Jul 27 16:19:17.943837 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:17.943819    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:19:18.447871 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:18.447837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" event={"ID":"0621648c-28a5-416b-895e-898baf34a3ae","Type":"ContainerStarted","Data":"d046e4f2557ec47a66721b74acdae4d7f08cc25bf17181fbebe37f5685d7b07d"}
Jul 27 16:19:20.459844 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:20.459807    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" event={"ID":"0621648c-28a5-416b-895e-898baf34a3ae","Type":"ContainerStarted","Data":"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"}
Jul 27 16:19:20.460227 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:20.459886    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:20.478728 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:20.478643    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" podStartSLOduration=2.067343933 podStartE2EDuration="4.47862651s" totalImagesPullingTime="2.411282577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:19:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:19:20.477671843 +0000 UTC m=+1856.391728392" watchObservedRunningTime="2026-07-27 16:19:20.47862651 +0000 UTC m=+1856.392683063"
Jul 27 16:19:24.097698 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:24.097663    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:19:26.469870 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:26.469835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:52.877860 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:52.877776    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:52.883575 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:52.877984    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" podUID="0621648c-28a5-416b-895e-898baf34a3ae" containerName="maas-api" containerID="cri-o://8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260" gracePeriod=30
Jul 27 16:19:53.121962 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.121937    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:53.220665 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.220637    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca\") pod \"0621648c-28a5-416b-895e-898baf34a3ae\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") "
Jul 27 16:19:53.220665 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.220670    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") pod \"0621648c-28a5-416b-895e-898baf34a3ae\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") "
Jul 27 16:19:53.220913 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.220686    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47xcj\" (UniqueName: \"kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj\") pod \"0621648c-28a5-416b-895e-898baf34a3ae\" (UID: \"0621648c-28a5-416b-895e-898baf34a3ae\") "
Jul 27 16:19:53.221063 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.221040    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0621648c-28a5-416b-895e-898baf34a3ae" (UID: "0621648c-28a5-416b-895e-898baf34a3ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:19:53.222900 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.222875    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj" (OuterVolumeSpecName: "kube-api-access-47xcj") pod "0621648c-28a5-416b-895e-898baf34a3ae" (UID: "0621648c-28a5-416b-895e-898baf34a3ae"). InnerVolumeSpecName "kube-api-access-47xcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:19:53.222991 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.222903    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0621648c-28a5-416b-895e-898baf34a3ae" (UID: "0621648c-28a5-416b-895e-898baf34a3ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:19:53.321475 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.321437    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0621648c-28a5-416b-895e-898baf34a3ae-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:19:53.321475 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.321470    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0621648c-28a5-416b-895e-898baf34a3ae-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:19:53.321475 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.321483    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47xcj\" (UniqueName: \"kubernetes.io/projected/0621648c-28a5-416b-895e-898baf34a3ae-kube-api-access-47xcj\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:19:53.604091 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.603997    2576 generic.go:356] "Generic (PLEG): container finished" podID="0621648c-28a5-416b-895e-898baf34a3ae" containerID="8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260" exitCode=0
Jul 27 16:19:53.604091 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.604042    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"
Jul 27 16:19:53.604294 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.604084    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" event={"ID":"0621648c-28a5-416b-895e-898baf34a3ae","Type":"ContainerDied","Data":"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"}
Jul 27 16:19:53.604294 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.604156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt" event={"ID":"0621648c-28a5-416b-895e-898baf34a3ae","Type":"ContainerDied","Data":"d046e4f2557ec47a66721b74acdae4d7f08cc25bf17181fbebe37f5685d7b07d"}
Jul 27 16:19:53.604294 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.604179    2576 scope.go:117] "RemoveContainer" containerID="8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"
Jul 27 16:19:53.612814 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.612796    2576 scope.go:117] "RemoveContainer" containerID="8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"
Jul 27 16:19:53.613053 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:19:53.613034    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260\": container with ID starting with 8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260 not found: ID does not exist" containerID="8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"
Jul 27 16:19:53.613187 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.613061    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260"} err="failed to get container status \"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260\": rpc error: code = NotFound desc = could not find container \"8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260\": container with ID starting with 8be3e84e1247938f164b9245c452e4259048a14c8a87ceb394b410ad2e7bd260 not found: ID does not exist"
Jul 27 16:19:53.626009 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.625987    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:53.629868 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:53.629847    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9ab13729-758656bbf6-9wzgt"]
Jul 27 16:19:54.652054 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:54.652018    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0621648c-28a5-416b-895e-898baf34a3ae" path="/var/lib/kubelet/pods/0621648c-28a5-416b-895e-898baf34a3ae/volumes"
Jul 27 16:19:55.898538 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:19:55.898501    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:20:01.237116 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.237060    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:01.237663 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.237638    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0621648c-28a5-416b-895e-898baf34a3ae" containerName="maas-api"
Jul 27 16:20:01.237663 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.237655    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0621648c-28a5-416b-895e-898baf34a3ae" containerName="maas-api"
Jul 27 16:20:01.237769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.237744    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0621648c-28a5-416b-895e-898baf34a3ae" containerName="maas-api"
Jul 27 16:20:01.240928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.240907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.243965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.243939    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ed0533f9-serving-cert\""
Jul 27 16:20:01.244094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.243939    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:20:01.244283 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.244245    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:20:01.244929 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.244908    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:20:01.249454 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.249080    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:01.277593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.277554    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfvh\" (UniqueName: \"kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.277593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.277596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.277830 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.277683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.378732 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.378691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfvh\" (UniqueName: \"kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.378910 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.378740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.378910 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.378827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.378996 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:01.378911    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-serving-cert: secret "maas-api-e2e-migrate-ed0533f9-serving-cert" not found
Jul 27 16:20:01.378996 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:01.378994    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls podName:3d5fee4c-f409-40cb-b996-58ce3617511e nodeName:}" failed. No retries permitted until 2026-07-27 16:20:01.878975936 +0000 UTC m=+1897.793032463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls") pod "maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" (UID: "3d5fee4c-f409-40cb-b996-58ce3617511e") : secret "maas-api-e2e-migrate-ed0533f9-serving-cert" not found
Jul 27 16:20:01.379473 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.379456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.390357 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.390330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfvh\" (UniqueName: \"kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.883234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:01.883189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:01.883410 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:01.883318    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-serving-cert: secret "maas-api-e2e-migrate-ed0533f9-serving-cert" not found
Jul 27 16:20:01.883410 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:01.883393    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls podName:3d5fee4c-f409-40cb-b996-58ce3617511e nodeName:}" failed. No retries permitted until 2026-07-27 16:20:02.883377488 +0000 UTC m=+1898.797434015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls") pod "maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" (UID: "3d5fee4c-f409-40cb-b996-58ce3617511e") : secret "maas-api-e2e-migrate-ed0533f9-serving-cert" not found
Jul 27 16:20:02.891645 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:02.891598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:02.894139 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:02.894093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:03.052858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:03.052813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:03.183467 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:03.183442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:03.185511 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:20:03.185490    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5fee4c_f409_40cb_b996_58ce3617511e.slice/crio-c861e3c6c3bf3a2079dee80e3d4b66af2484ad9b87966b7aa4104f64b587b92a WatchSource:0}: Error finding container c861e3c6c3bf3a2079dee80e3d4b66af2484ad9b87966b7aa4104f64b587b92a: Status 404 returned error can't find the container with id c861e3c6c3bf3a2079dee80e3d4b66af2484ad9b87966b7aa4104f64b587b92a
Jul 27 16:20:03.652137 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:03.652082    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" event={"ID":"3d5fee4c-f409-40cb-b996-58ce3617511e","Type":"ContainerStarted","Data":"c861e3c6c3bf3a2079dee80e3d4b66af2484ad9b87966b7aa4104f64b587b92a"}
Jul 27 16:20:04.657152 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:04.657119    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" event={"ID":"3d5fee4c-f409-40cb-b996-58ce3617511e","Type":"ContainerStarted","Data":"ca2f95b93c14d3956e91159c187be1957ca03abf1d36fadf0f9e016b9e33ab38"}
Jul 27 16:20:04.657543 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:04.657227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:04.678260 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:04.678203    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" podStartSLOduration=3.011252813 podStartE2EDuration="3.678187502s" totalImagesPullingTime="666.934689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:20:04.676536885 +0000 UTC m=+1900.590593433" watchObservedRunningTime="2026-07-27 16:20:04.678187502 +0000 UTC m=+1900.592244028"
Jul 27 16:20:07.092831 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:07.092792    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:20:10.667370 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:10.667339    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:36.641842 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.641807    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:36.642613 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.642047    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" podUID="3d5fee4c-f409-40cb-b996-58ce3617511e" containerName="maas-api" containerID="cri-o://ca2f95b93c14d3956e91159c187be1957ca03abf1d36fadf0f9e016b9e33ab38" gracePeriod=30
Jul 27 16:20:36.797337 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.797296    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d5fee4c-f409-40cb-b996-58ce3617511e" containerID="ca2f95b93c14d3956e91159c187be1957ca03abf1d36fadf0f9e016b9e33ab38" exitCode=0
Jul 27 16:20:36.797501 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.797377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" event={"ID":"3d5fee4c-f409-40cb-b996-58ce3617511e","Type":"ContainerDied","Data":"ca2f95b93c14d3956e91159c187be1957ca03abf1d36fadf0f9e016b9e33ab38"}
Jul 27 16:20:36.898988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.898938    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:36.997643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.997597    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpfvh\" (UniqueName: \"kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh\") pod \"3d5fee4c-f409-40cb-b996-58ce3617511e\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") "
Jul 27 16:20:36.997811 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.997658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca\") pod \"3d5fee4c-f409-40cb-b996-58ce3617511e\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") "
Jul 27 16:20:36.997811 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.997680    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") pod \"3d5fee4c-f409-40cb-b996-58ce3617511e\" (UID: \"3d5fee4c-f409-40cb-b996-58ce3617511e\") "
Jul 27 16:20:36.998094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.998065    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3d5fee4c-f409-40cb-b996-58ce3617511e" (UID: "3d5fee4c-f409-40cb-b996-58ce3617511e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:20:36.999685 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.999661    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3d5fee4c-f409-40cb-b996-58ce3617511e" (UID: "3d5fee4c-f409-40cb-b996-58ce3617511e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:20:36.999731 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:36.999664    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh" (OuterVolumeSpecName: "kube-api-access-hpfvh") pod "3d5fee4c-f409-40cb-b996-58ce3617511e" (UID: "3d5fee4c-f409-40cb-b996-58ce3617511e"). InnerVolumeSpecName "kube-api-access-hpfvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:20:37.098534 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.098502    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfvh\" (UniqueName: \"kubernetes.io/projected/3d5fee4c-f409-40cb-b996-58ce3617511e-kube-api-access-hpfvh\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:20:37.098534 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.098528    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d5fee4c-f409-40cb-b996-58ce3617511e-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:20:37.098534 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.098538    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d5fee4c-f409-40cb-b996-58ce3617511e-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:20:37.804183 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.804151    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8" event={"ID":"3d5fee4c-f409-40cb-b996-58ce3617511e","Type":"ContainerDied","Data":"c861e3c6c3bf3a2079dee80e3d4b66af2484ad9b87966b7aa4104f64b587b92a"}
Jul 27 16:20:37.804611 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.804217    2576 scope.go:117] "RemoveContainer" containerID="ca2f95b93c14d3956e91159c187be1957ca03abf1d36fadf0f9e016b9e33ab38"
Jul 27 16:20:37.804611 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.804163    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"
Jul 27 16:20:37.828323 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.828294    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:37.834255 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:37.834230    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ed0533f9-5f69ffcc7d-s85d8"]
Jul 27 16:20:38.654068 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:38.654038    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d5fee4c-f409-40cb-b996-58ce3617511e" path="/var/lib/kubelet/pods/3d5fee4c-f409-40cb-b996-58ce3617511e/volumes"
Jul 27 16:20:39.712430 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:39.712393    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:20:44.643749 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.643708    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:20:44.644286 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.644265    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d5fee4c-f409-40cb-b996-58ce3617511e" containerName="maas-api"
Jul 27 16:20:44.644345 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.644291    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d5fee4c-f409-40cb-b996-58ce3617511e" containerName="maas-api"
Jul 27 16:20:44.644418 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.644406    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d5fee4c-f409-40cb-b996-58ce3617511e" containerName="maas-api"
Jul 27 16:20:44.649460 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.649428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.652907 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.652885    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:20:44.653035 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.652920    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:20:44.653353 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.653335    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-09433aa9-serving-cert\""
Jul 27 16:20:44.654313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.654295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:20:44.660706 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.660684    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:20:44.768708 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.768676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbmw4\" (UniqueName: \"kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.768839 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.768755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.768839 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.768807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.869474 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.869441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.869678 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.869535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbmw4\" (UniqueName: \"kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.869678 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.869601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.869846 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:44.869708    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-serving-cert: secret "maas-api-e2e-ait-09433aa9-serving-cert" not found
Jul 27 16:20:44.869846 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:20:44.869778    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls podName:7f853072-20ed-497d-af99-b486d2b46fc9 nodeName:}" failed. No retries permitted until 2026-07-27 16:20:45.369756289 +0000 UTC m=+1941.283812836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls") pod "maas-api-e2e-ait-09433aa9-7f686664c6-578mj" (UID: "7f853072-20ed-497d-af99-b486d2b46fc9") : secret "maas-api-e2e-ait-09433aa9-serving-cert" not found
Jul 27 16:20:44.870171 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.870150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:44.879535 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:44.879513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbmw4\" (UniqueName: \"kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:45.373681 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:45.373646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:45.375997 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:45.375979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") pod \"maas-api-e2e-ait-09433aa9-7f686664c6-578mj\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:45.563787 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:45.563753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:45.736039 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:45.736009    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:20:45.737503 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:20:45.737476    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f853072_20ed_497d_af99_b486d2b46fc9.slice/crio-58d4152d1bad1adb1dbb17d36a6e783a676a228e4946d4458153b25e4f089487 WatchSource:0}: Error finding container 58d4152d1bad1adb1dbb17d36a6e783a676a228e4946d4458153b25e4f089487: Status 404 returned error can't find the container with id 58d4152d1bad1adb1dbb17d36a6e783a676a228e4946d4458153b25e4f089487
Jul 27 16:20:45.840165 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:45.840129    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" event={"ID":"7f853072-20ed-497d-af99-b486d2b46fc9","Type":"ContainerStarted","Data":"58d4152d1bad1adb1dbb17d36a6e783a676a228e4946d4458153b25e4f089487"}
Jul 27 16:20:46.846465 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:46.846421    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" event={"ID":"7f853072-20ed-497d-af99-b486d2b46fc9","Type":"ContainerStarted","Data":"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"}
Jul 27 16:20:46.846845 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:46.846558    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:20:46.880602 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:46.880514    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" podStartSLOduration=1.993173089 podStartE2EDuration="2.880500361s" totalImagesPullingTime="887.327272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:20:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:20:46.879242366 +0000 UTC m=+1942.793298916" watchObservedRunningTime="2026-07-27 16:20:46.880500361 +0000 UTC m=+1942.794556909"
Jul 27 16:20:47.911046 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:47.911014    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:20:52.857089 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:20:52.857061    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:21:20.745800 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:20.745719    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:21:20.746276 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:20.745946    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" podUID="7f853072-20ed-497d-af99-b486d2b46fc9" containerName="maas-api" containerID="cri-o://1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80" gracePeriod=30
Jul 27 16:21:20.996214 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:20.996161    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:21:21.007565 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.007539    2576 generic.go:356] "Generic (PLEG): container finished" podID="7f853072-20ed-497d-af99-b486d2b46fc9" containerID="1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80" exitCode=0
Jul 27 16:21:21.007693 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.007579    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"
Jul 27 16:21:21.007693 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.007603    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" event={"ID":"7f853072-20ed-497d-af99-b486d2b46fc9","Type":"ContainerDied","Data":"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"}
Jul 27 16:21:21.007693 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.007660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj" event={"ID":"7f853072-20ed-497d-af99-b486d2b46fc9","Type":"ContainerDied","Data":"58d4152d1bad1adb1dbb17d36a6e783a676a228e4946d4458153b25e4f089487"}
Jul 27 16:21:21.007693 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.007683    2576 scope.go:117] "RemoveContainer" containerID="1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"
Jul 27 16:21:21.016625 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.016610    2576 scope.go:117] "RemoveContainer" containerID="1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"
Jul 27 16:21:21.016924 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:21:21.016892    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80\": container with ID starting with 1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80 not found: ID does not exist" containerID="1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"
Jul 27 16:21:21.017031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.016932    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80"} err="failed to get container status \"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80\": rpc error: code = NotFound desc = could not find container \"1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80\": container with ID starting with 1961a3467e0d39eb2c7dc080f8b3cb0875dff4579e060ef03d0378a56cd1fd80 not found: ID does not exist"
Jul 27 16:21:21.041374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.041350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbmw4\" (UniqueName: \"kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4\") pod \"7f853072-20ed-497d-af99-b486d2b46fc9\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") "
Jul 27 16:21:21.041484 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.041443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") pod \"7f853072-20ed-497d-af99-b486d2b46fc9\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") "
Jul 27 16:21:21.041553 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.041498    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca\") pod \"7f853072-20ed-497d-af99-b486d2b46fc9\" (UID: \"7f853072-20ed-497d-af99-b486d2b46fc9\") "
Jul 27 16:21:21.041913 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.041884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f853072-20ed-497d-af99-b486d2b46fc9" (UID: "7f853072-20ed-497d-af99-b486d2b46fc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:21.043433 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.043405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4" (OuterVolumeSpecName: "kube-api-access-sbmw4") pod "7f853072-20ed-497d-af99-b486d2b46fc9" (UID: "7f853072-20ed-497d-af99-b486d2b46fc9"). InnerVolumeSpecName "kube-api-access-sbmw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:21:21.043546 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.043511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f853072-20ed-497d-af99-b486d2b46fc9" (UID: "7f853072-20ed-497d-af99-b486d2b46fc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:21:21.143163 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.143136    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f853072-20ed-497d-af99-b486d2b46fc9-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:21.143163 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.143162    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbmw4\" (UniqueName: \"kubernetes.io/projected/7f853072-20ed-497d-af99-b486d2b46fc9-kube-api-access-sbmw4\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:21.143311 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.143175    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f853072-20ed-497d-af99-b486d2b46fc9-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:21.333167 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.333118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:21:21.335690 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:21.335661    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09433aa9-7f686664c6-578mj"]
Jul 27 16:21:22.652180 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:22.652143    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f853072-20ed-497d-af99-b486d2b46fc9" path="/var/lib/kubelet/pods/7f853072-20ed-497d-af99-b486d2b46fc9/volumes"
Jul 27 16:21:23.802829 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:23.802791    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:21:29.714159 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.714092    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:29.714643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.714440    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f853072-20ed-497d-af99-b486d2b46fc9" containerName="maas-api"
Jul 27 16:21:29.714643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.714452    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f853072-20ed-497d-af99-b486d2b46fc9" containerName="maas-api"
Jul 27 16:21:29.714643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.714520    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f853072-20ed-497d-af99-b486d2b46fc9" containerName="maas-api"
Jul 27 16:21:29.716829 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.716809    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.721527 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.721504    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0a85b590-serving-cert\""
Jul 27 16:21:29.721527 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.721517    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:21:29.722027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.722006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:21:29.726787 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.726766    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:21:29.737742 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.737707    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:29.825871 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.825842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4mn\" (UniqueName: \"kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.826041 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.825902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.826093 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.826059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.927119 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.927075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.927295 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.927181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.927295 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.927216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4mn\" (UniqueName: \"kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.927794 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.927772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.929423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.929403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:29.942602 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:29.942583    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4mn\" (UniqueName: \"kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn\") pod \"maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:30.027478 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:30.027405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:30.163444 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:30.163419    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:30.165007 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:21:30.164979    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fb9876_dea4_4913_b76c_b9a4105149ae.slice/crio-eff451649f0ce97e384fc1b7ccedcd78cb5d1d482bdc312758d7cfab6eada24c WatchSource:0}: Error finding container eff451649f0ce97e384fc1b7ccedcd78cb5d1d482bdc312758d7cfab6eada24c: Status 404 returned error can't find the container with id eff451649f0ce97e384fc1b7ccedcd78cb5d1d482bdc312758d7cfab6eada24c
Jul 27 16:21:31.058404 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:31.058366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" event={"ID":"18fb9876-dea4-4913-b76c-b9a4105149ae","Type":"ContainerStarted","Data":"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"}
Jul 27 16:21:31.058802 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:31.058422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" event={"ID":"18fb9876-dea4-4913-b76c-b9a4105149ae","Type":"ContainerStarted","Data":"eff451649f0ce97e384fc1b7ccedcd78cb5d1d482bdc312758d7cfab6eada24c"}
Jul 27 16:21:31.058802 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:31.058468    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:31.083643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:31.083592    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" podStartSLOduration=1.634367382 podStartE2EDuration="2.083576989s" totalImagesPullingTime="449.209607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:21:31.082536276 +0000 UTC m=+1986.996592823" watchObservedRunningTime="2026-07-27 16:21:31.083576989 +0000 UTC m=+1986.997633538"
Jul 27 16:21:33.310090 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:33.310059    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:21:37.069322 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:37.069296    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:50.764718 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:50.764679    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:50.765196 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:50.764900    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" podUID="18fb9876-dea4-4913-b76c-b9a4105149ae" containerName="maas-api" containerID="cri-o://daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1" gracePeriod=30
Jul 27 16:21:51.001605 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.001582    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:51.116646 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.116571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls\") pod \"18fb9876-dea4-4913-b76c-b9a4105149ae\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") "
Jul 27 16:21:51.116805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.116665    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db4mn\" (UniqueName: \"kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn\") pod \"18fb9876-dea4-4913-b76c-b9a4105149ae\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") "
Jul 27 16:21:51.116805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.116721    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca\") pod \"18fb9876-dea4-4913-b76c-b9a4105149ae\" (UID: \"18fb9876-dea4-4913-b76c-b9a4105149ae\") "
Jul 27 16:21:51.117088 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.117053    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18fb9876-dea4-4913-b76c-b9a4105149ae" (UID: "18fb9876-dea4-4913-b76c-b9a4105149ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:21:51.118718 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.118691    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn" (OuterVolumeSpecName: "kube-api-access-db4mn") pod "18fb9876-dea4-4913-b76c-b9a4105149ae" (UID: "18fb9876-dea4-4913-b76c-b9a4105149ae"). InnerVolumeSpecName "kube-api-access-db4mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:21:51.118718 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.118696    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18fb9876-dea4-4913-b76c-b9a4105149ae" (UID: "18fb9876-dea4-4913-b76c-b9a4105149ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:21:51.150822 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.150788    2576 generic.go:356] "Generic (PLEG): container finished" podID="18fb9876-dea4-4913-b76c-b9a4105149ae" containerID="daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1" exitCode=0
Jul 27 16:21:51.150947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.150845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" event={"ID":"18fb9876-dea4-4913-b76c-b9a4105149ae","Type":"ContainerDied","Data":"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"}
Jul 27 16:21:51.150947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.150886    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8" event={"ID":"18fb9876-dea4-4913-b76c-b9a4105149ae","Type":"ContainerDied","Data":"eff451649f0ce97e384fc1b7ccedcd78cb5d1d482bdc312758d7cfab6eada24c"}
Jul 27 16:21:51.150947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.150846    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"
Jul 27 16:21:51.150947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.150908    2576 scope.go:117] "RemoveContainer" containerID="daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"
Jul 27 16:21:51.160205 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.160182    2576 scope.go:117] "RemoveContainer" containerID="daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"
Jul 27 16:21:51.160573 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:21:51.160532    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1\": container with ID starting with daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1 not found: ID does not exist" containerID="daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"
Jul 27 16:21:51.160705 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.160569    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1"} err="failed to get container status \"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1\": rpc error: code = NotFound desc = could not find container \"daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1\": container with ID starting with daf3c7270ffb7056323550b0b48509b0d6173192ba54eb0cc345f99df8efa4d1 not found: ID does not exist"
Jul 27 16:21:51.178664 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.178640    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:51.184744 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.184724    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0a85b590-85b5bd975c-lqwm8"]
Jul 27 16:21:51.217523 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.217497    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18fb9876-dea4-4913-b76c-b9a4105149ae-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:51.217523 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.217521    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4mn\" (UniqueName: \"kubernetes.io/projected/18fb9876-dea4-4913-b76c-b9a4105149ae-kube-api-access-db4mn\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:51.217665 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:51.217544    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18fb9876-dea4-4913-b76c-b9a4105149ae-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:21:52.652309 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:52.652274    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fb9876-dea4-4913-b76c-b9a4105149ae" path="/var/lib/kubelet/pods/18fb9876-dea4-4913-b76c-b9a4105149ae/volumes"
Jul 27 16:21:53.893851 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:21:53.893816    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:23:22.800893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:22.800812    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:23:24.628593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:24.628557    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:23:24.636841 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:24.636815    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:23:32.926577 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:32.926542    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:23:40.107020 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.106987    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:23:40.107423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.107349    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fb9876-dea4-4913-b76c-b9a4105149ae" containerName="maas-api"
Jul 27 16:23:40.107423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.107362    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fb9876-dea4-4913-b76c-b9a4105149ae" containerName="maas-api"
Jul 27 16:23:40.107423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.107420    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fb9876-dea4-4913-b76c-b9a4105149ae" containerName="maas-api"
Jul 27 16:23:40.110525 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.110507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.113364 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.113342    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:23:40.114223 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.114203    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:23:40.114343 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.114228    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:23:40.114343 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.114207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-43178a25-serving-cert\""
Jul 27 16:23:40.119935 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.119912    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:23:40.254912 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.254880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.255135 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.254961    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vbk\" (UniqueName: \"kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.255135 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.255017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.355769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.355726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.355966 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.355799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vbk\" (UniqueName: \"kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.355966 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.355848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.356529 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.356504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.358251 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.358198    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.364704 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.364679    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vbk\" (UniqueName: \"kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk\") pod \"maas-api-e2e-mt-43178a25-ccf694c88-2s48s\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.423205 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.423180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:40.561330 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.561306    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:23:40.563527 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:23:40.563500    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode142530f_32a4_47b3_be20_d2bd640695b7.slice/crio-bcbb647d958747701fa31d65da45d69e74b2101e2d84315b588655742a1aa295 WatchSource:0}: Error finding container bcbb647d958747701fa31d65da45d69e74b2101e2d84315b588655742a1aa295: Status 404 returned error can't find the container with id bcbb647d958747701fa31d65da45d69e74b2101e2d84315b588655742a1aa295
Jul 27 16:23:40.634541 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:40.634463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" event={"ID":"e142530f-32a4-47b3-be20-d2bd640695b7","Type":"ContainerStarted","Data":"bcbb647d958747701fa31d65da45d69e74b2101e2d84315b588655742a1aa295"}
Jul 27 16:23:41.642201 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:41.642097    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" event={"ID":"e142530f-32a4-47b3-be20-d2bd640695b7","Type":"ContainerStarted","Data":"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"}
Jul 27 16:23:41.642601 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:41.642250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:23:41.662377 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:41.662332    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" podStartSLOduration=1.012423948 podStartE2EDuration="1.662318664s" totalImagesPullingTime="649.894716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:23:41.660033795 +0000 UTC m=+2117.574090369" watchObservedRunningTime="2026-07-27 16:23:41.662318664 +0000 UTC m=+2117.576375211"
Jul 27 16:23:45.596446 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:45.596400    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:23:47.652687 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:23:47.652655    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:24:02.421228 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.421196    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:24:02.421669 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.421384    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" containerName="maas-api" containerID="cri-o://46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc" gracePeriod=30
Jul 27 16:24:02.674825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.674759    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:24:02.733064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.733029    2576 generic.go:356] "Generic (PLEG): container finished" podID="e142530f-32a4-47b3-be20-d2bd640695b7" containerID="46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc" exitCode=0
Jul 27 16:24:02.733064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.733066    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"
Jul 27 16:24:02.733314 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.733124    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" event={"ID":"e142530f-32a4-47b3-be20-d2bd640695b7","Type":"ContainerDied","Data":"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"}
Jul 27 16:24:02.733314 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.733188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" event={"ID":"e142530f-32a4-47b3-be20-d2bd640695b7","Type":"ContainerDied","Data":"bcbb647d958747701fa31d65da45d69e74b2101e2d84315b588655742a1aa295"}
Jul 27 16:24:02.733314 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.733216    2576 scope.go:117] "RemoveContainer" containerID="46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"
Jul 27 16:24:02.743057 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.743038    2576 scope.go:117] "RemoveContainer" containerID="46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"
Jul 27 16:24:02.743353 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:24:02.743330    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc\": container with ID starting with 46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc not found: ID does not exist" containerID="46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"
Jul 27 16:24:02.743413 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.743361    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc"} err="failed to get container status \"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc\": rpc error: code = NotFound desc = could not find container \"46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc\": container with ID starting with 46e0670482975b74f9280b7d87d69acd07eb39ba49bec723507fc9500d06adfc not found: ID does not exist"
Jul 27 16:24:02.748668 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.748650    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls\") pod \"e142530f-32a4-47b3-be20-d2bd640695b7\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") "
Jul 27 16:24:02.748763 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.748676    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca\") pod \"e142530f-32a4-47b3-be20-d2bd640695b7\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") "
Jul 27 16:24:02.748763 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.748722    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5vbk\" (UniqueName: \"kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk\") pod \"e142530f-32a4-47b3-be20-d2bd640695b7\" (UID: \"e142530f-32a4-47b3-be20-d2bd640695b7\") "
Jul 27 16:24:02.749091 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.749063    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e142530f-32a4-47b3-be20-d2bd640695b7" (UID: "e142530f-32a4-47b3-be20-d2bd640695b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:24:02.750963 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.750938    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk" (OuterVolumeSpecName: "kube-api-access-m5vbk") pod "e142530f-32a4-47b3-be20-d2bd640695b7" (UID: "e142530f-32a4-47b3-be20-d2bd640695b7"). InnerVolumeSpecName "kube-api-access-m5vbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:24:02.751219 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.751196    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e142530f-32a4-47b3-be20-d2bd640695b7" (UID: "e142530f-32a4-47b3-be20-d2bd640695b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:24:02.849552 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.849513    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e142530f-32a4-47b3-be20-d2bd640695b7-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:24:02.849552 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.849545    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e142530f-32a4-47b3-be20-d2bd640695b7-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:24:02.849552 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:02.849557    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vbk\" (UniqueName: \"kubernetes.io/projected/e142530f-32a4-47b3-be20-d2bd640695b7-kube-api-access-m5vbk\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:24:03.056116 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:03.056080    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:24:03.060606 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:03.060582    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s"]
Jul 27 16:24:04.652047 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:04.652012    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" path="/var/lib/kubelet/pods/e142530f-32a4-47b3-be20-d2bd640695b7/volumes"
Jul 27 16:24:05.303966 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:05.303934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:24:05.648314 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:05.648219    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-43178a25-ccf694c88-2s48s" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.39:8443/health\": context deadline exceeded"
Jul 27 16:24:37.340231 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.340193    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:24:37.340762 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.340673    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" containerName="maas-api"
Jul 27 16:24:37.340762 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.340688    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" containerName="maas-api"
Jul 27 16:24:37.340837 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.340793    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e142530f-32a4-47b3-be20-d2bd640695b7" containerName="maas-api"
Jul 27 16:24:37.344046 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.344028    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.347139 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.347117    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:24:37.348025 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.348001    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e1cdb3d5-serving-cert\""
Jul 27 16:24:37.348159 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.348036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:24:37.348159 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.348005    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:24:37.353642 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.353606    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:24:37.452180 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.452148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.452354 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.452191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxv2\" (UniqueName: \"kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.452354 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.452230    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.552824 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.552790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.553002 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.552843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxv2\" (UniqueName: \"kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.553002 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.552876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.553002 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:24:37.552898    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-serving-cert: secret "maas-api-e2e-mt-e1cdb3d5-serving-cert" not found
Jul 27 16:24:37.553002 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:24:37.552974    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls podName:82cd93c0-6ddc-467f-a0a2-d42c4674a2ef nodeName:}" failed. No retries permitted until 2026-07-27 16:24:38.052957369 +0000 UTC m=+2173.967013896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls") pod "maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" (UID: "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef") : secret "maas-api-e2e-mt-e1cdb3d5-serving-cert" not found
Jul 27 16:24:37.553599 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.553576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:37.585738 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:37.585710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxv2\" (UniqueName: \"kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:38.057509 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.057463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:38.060443 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.060409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") pod \"maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:38.255041 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.255009    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:38.394620 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.394583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:24:38.398801 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.398773    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:24:38.902594 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.902555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" event={"ID":"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef","Type":"ContainerStarted","Data":"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"}
Jul 27 16:24:38.902773 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.902619    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" event={"ID":"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef","Type":"ContainerStarted","Data":"49d821d9c68944ac2f5aed21ab11c45d2f5a8a840bf9eda17e060d6a37b7cdae"}
Jul 27 16:24:38.902773 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.902764    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:38.926407 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:38.926361    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" podStartSLOduration=1.566477997 podStartE2EDuration="1.926340236s" totalImagesPullingTime="359.862239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:24:38.925003859 +0000 UTC m=+2174.839060408" watchObservedRunningTime="2026-07-27 16:24:38.926340236 +0000 UTC m=+2174.840396786"
Jul 27 16:24:43.402902 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:43.402868    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:24:44.912024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:44.911996    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:24:58.905805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.905770    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:24:58.909077 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.909059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:58.911755 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.911731    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1cdb3d5\"/\"openshift-service-ca.crt\""
Jul 27 16:24:58.912950 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.912926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1cdb3d5\"/\"e2e-lifecycle-model-e1cdb3d5-kserve-self-signed-certs\""
Jul 27 16:24:58.913067 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.912963    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e1cdb3d5\"/\"kube-root-ca.crt\""
Jul 27 16:24:58.913067 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.912936    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e1cdb3d5\"/\"default-dockercfg-svb52\""
Jul 27 16:24:58.919892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:58.919867    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:24:59.037847 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.037847 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037859    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.038075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.038075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.038075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgjwh\" (UniqueName: \"kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.038075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.037987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138411 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138566 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138566 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138689 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138689 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgjwh\" (UniqueName: \"kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138689 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.138922 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138897    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.139045 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.139045 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.138963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.140774 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.140748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.141073 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.141054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.151568 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.151544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgjwh\" (UniqueName: \"kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh\") pod \"e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") " pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.218545 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.218525    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:24:59.353494 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:24:59.353470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:24:59.355422 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:24:59.355389    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90c28e86_685d_465b_9c27_d9e02be16047.slice/crio-f6077af5c89f95605025dcd052258e0287bee17d20cf096e4ed1aff53c05b4de WatchSource:0}: Error finding container f6077af5c89f95605025dcd052258e0287bee17d20cf096e4ed1aff53c05b4de: Status 404 returned error can't find the container with id f6077af5c89f95605025dcd052258e0287bee17d20cf096e4ed1aff53c05b4de
Jul 27 16:25:00.002887 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:00.002850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" event={"ID":"90c28e86-685d-465b-9c27-d9e02be16047","Type":"ContainerStarted","Data":"ecf4f76d21b4afa66d6c1898949de44cb5f264aa4c9f4980591d5540f502d816"}
Jul 27 16:25:00.003280 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:00.002896    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" event={"ID":"90c28e86-685d-465b-9c27-d9e02be16047","Type":"ContainerStarted","Data":"f6077af5c89f95605025dcd052258e0287bee17d20cf096e4ed1aff53c05b4de"}
Jul 27 16:25:00.021974 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:00.021926    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" podStartSLOduration=2.021909254 podStartE2EDuration="2.021909254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:25:00.020505655 +0000 UTC m=+2195.934562202" watchObservedRunningTime="2026-07-27 16:25:00.021909254 +0000 UTC m=+2195.935965850"
Jul 27 16:25:02.601829 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:02.601789    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:25:09.218682 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:09.218653    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:25:09.219255 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:09.218702    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:25:09.231663 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:09.231639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:25:10.058765 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:10.058734    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:25:17.298526 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:17.298490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:25:35.594429 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:35.594391    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:25:37.065770 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.065714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:25:37.066326 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.065973    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" podUID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" containerName="maas-api" containerID="cri-o://1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934" gracePeriod=30
Jul 27 16:25:37.314731 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.314705    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:25:37.369198 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.369121    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") pod \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") "
Jul 27 16:25:37.369198 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.369166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca\") pod \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") "
Jul 27 16:25:37.369374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.369221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfxv2\" (UniqueName: \"kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2\") pod \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\" (UID: \"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef\") "
Jul 27 16:25:37.369537 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.369514    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" (UID: "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:25:37.371262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.371237    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" (UID: "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:25:37.371368 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.371345    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2" (OuterVolumeSpecName: "kube-api-access-dfxv2") pod "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" (UID: "82cd93c0-6ddc-467f-a0a2-d42c4674a2ef"). InnerVolumeSpecName "kube-api-access-dfxv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:25:37.470095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.470057    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfxv2\" (UniqueName: \"kubernetes.io/projected/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-kube-api-access-dfxv2\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:25:37.470095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.470088    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:25:37.470095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:37.470115    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:25:38.173035 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.173003    2576 generic.go:356] "Generic (PLEG): container finished" podID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" containerID="1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934" exitCode=0
Jul 27 16:25:38.173502 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.173048    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"
Jul 27 16:25:38.173502 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.173091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" event={"ID":"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef","Type":"ContainerDied","Data":"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"}
Jul 27 16:25:38.173502 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.173194    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h" event={"ID":"82cd93c0-6ddc-467f-a0a2-d42c4674a2ef","Type":"ContainerDied","Data":"49d821d9c68944ac2f5aed21ab11c45d2f5a8a840bf9eda17e060d6a37b7cdae"}
Jul 27 16:25:38.173502 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.173231    2576 scope.go:117] "RemoveContainer" containerID="1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"
Jul 27 16:25:38.181758 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.181742    2576 scope.go:117] "RemoveContainer" containerID="1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"
Jul 27 16:25:38.182007 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:25:38.181989    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934\": container with ID starting with 1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934 not found: ID does not exist" containerID="1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"
Jul 27 16:25:38.182064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.182016    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934"} err="failed to get container status \"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934\": rpc error: code = NotFound desc = could not find container \"1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934\": container with ID starting with 1e98ef73eb4cc5d6bc8bda8477995e36504215bea1bee1331aac0339cf6aa934 not found: ID does not exist"
Jul 27 16:25:38.195358 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.195333    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:25:38.197513 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.197493    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e1cdb3d5-75568bff58-8d26h"]
Jul 27 16:25:38.652242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:38.652212    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" path="/var/lib/kubelet/pods/82cd93c0-6ddc-467f-a0a2-d42c4674a2ef/volumes"
Jul 27 16:25:45.803085 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:45.803051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:25:50.146830 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:25:50.146794    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:26:05.166193 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.166135    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" podUID="90c28e86-685d-465b-9c27-d9e02be16047" containerName="main" containerID="cri-o://ecf4f76d21b4afa66d6c1898949de44cb5f264aa4c9f4980591d5540f502d816" gracePeriod=45
Jul 27 16:26:05.293617 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.293585    2576 generic.go:356] "Generic (PLEG): container finished" podID="90c28e86-685d-465b-9c27-d9e02be16047" containerID="ecf4f76d21b4afa66d6c1898949de44cb5f264aa4c9f4980591d5540f502d816" exitCode=0
Jul 27 16:26:05.293793 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.293643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" event={"ID":"90c28e86-685d-465b-9c27-d9e02be16047","Type":"ContainerDied","Data":"ecf4f76d21b4afa66d6c1898949de44cb5f264aa4c9f4980591d5540f502d816"}
Jul 27 16:26:05.422272 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.422211    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:26:05.521877 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.521841    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522017 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.521910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgjwh\" (UniqueName: \"kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522017 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.521961    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522017 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522001    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522195 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522195 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522042    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache\") pod \"90c28e86-685d-465b-9c27-d9e02be16047\" (UID: \"90c28e86-685d-465b-9c27-d9e02be16047\") "
Jul 27 16:26:05.522376 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522341    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home" (OuterVolumeSpecName: "home") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:26:05.522500 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522337    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:26:05.522500 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.522390    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache" (OuterVolumeSpecName: "model-cache") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:26:05.524145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.524113    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh" (OuterVolumeSpecName: "kube-api-access-jgjwh") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "kube-api-access-jgjwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:26:05.524237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.524165    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm" (OuterVolumeSpecName: "dshm") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:26:05.524237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.524168    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90c28e86-685d-465b-9c27-d9e02be16047" (UID: "90c28e86-685d-465b-9c27-d9e02be16047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:26:05.622699 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622673    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgjwh\" (UniqueName: \"kubernetes.io/projected/90c28e86-685d-465b-9c27-d9e02be16047-kube-api-access-jgjwh\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:05.622808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622718    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:05.622808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622730    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90c28e86-685d-465b-9c27-d9e02be16047-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:05.622808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622738    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:05.622808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622746    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:05.622808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:05.622753    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90c28e86-685d-465b-9c27-d9e02be16047-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:26:06.299406 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.299378    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"
Jul 27 16:26:06.299869 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.299377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m" event={"ID":"90c28e86-685d-465b-9c27-d9e02be16047","Type":"ContainerDied","Data":"f6077af5c89f95605025dcd052258e0287bee17d20cf096e4ed1aff53c05b4de"}
Jul 27 16:26:06.299869 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.299509    2576 scope.go:117] "RemoveContainer" containerID="ecf4f76d21b4afa66d6c1898949de44cb5f264aa4c9f4980591d5540f502d816"
Jul 27 16:26:06.324153 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.324128    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:26:06.330267 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.330246    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e1cdb3d5/e2e-lifecycle-model-e1cdb3d5-kserve-756d7df5dc-gtm2m"]
Jul 27 16:26:06.652503 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:06.652428    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90c28e86-685d-465b-9c27-d9e02be16047" path="/var/lib/kubelet/pods/90c28e86-685d-465b-9c27-d9e02be16047/volumes"
Jul 27 16:26:38.111299 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:38.111258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:26:48.517203 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:26:48.517168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:27:35.320204 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320132    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320484    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90c28e86-685d-465b-9c27-d9e02be16047" containerName="main"
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320496    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90c28e86-685d-465b-9c27-d9e02be16047" containerName="main"
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320511    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" containerName="maas-api"
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320517    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" containerName="maas-api"
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320573    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90c28e86-685d-465b-9c27-d9e02be16047" containerName="main"
Jul 27 16:27:35.320650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.320582    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cd93c0-6ddc-467f-a0a2-d42c4674a2ef" containerName="maas-api"
Jul 27 16:27:35.323477 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.323462    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.326155 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.326132    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:27:35.326291 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.326201    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8cec93-serving-cert\""
Jul 27 16:27:35.327234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.327217    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:27:35.327234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.327225    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:27:35.336484 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.336464    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:27:35.417879 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.417851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.418054 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.417886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzwwh\" (UniqueName: \"kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.418054 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.417975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.518793 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.518750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.518793 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.518794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzwwh\" (UniqueName: \"kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.519043 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.518872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.519043 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:27:35.519003    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-serving-cert: secret "maas-api-e2e-inf-a-8cec93-serving-cert" not found
Jul 27 16:27:35.519167 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:27:35.519087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls podName:eea96160-5763-4fd3-8144-32a4fb11f464 nodeName:}" failed. No retries permitted until 2026-07-27 16:27:36.019064593 +0000 UTC m=+2351.933121121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls") pod "maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" (UID: "eea96160-5763-4fd3-8144-32a4fb11f464") : secret "maas-api-e2e-inf-a-8cec93-serving-cert" not found
Jul 27 16:27:35.519594 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.519574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:35.530364 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:35.530340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzwwh\" (UniqueName: \"kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:36.023354 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:36.023318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:36.025898 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:36.025873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") pod \"maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:36.234268 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:36.234228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:36.369827 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:36.369801    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:27:36.706842 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:36.706805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" event={"ID":"eea96160-5763-4fd3-8144-32a4fb11f464","Type":"ContainerStarted","Data":"c811d518f1830d03c9ba32479de4d7771a41d51f9b7453719d1575772682a8d5"}
Jul 27 16:27:37.712808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:37.712772    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" event={"ID":"eea96160-5763-4fd3-8144-32a4fb11f464","Type":"ContainerStarted","Data":"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"}
Jul 27 16:27:37.713220 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:37.712877    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:37.730541 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:37.730494    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" podStartSLOduration=2.312617717 podStartE2EDuration="2.730478014s" totalImagesPullingTime="417.860297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:37.729143139 +0000 UTC m=+2353.643199699" watchObservedRunningTime="2026-07-27 16:27:37.730478014 +0000 UTC m=+2353.644534562"
Jul 27 16:27:40.209368 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:40.209334    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:27:43.310148 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.310110    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:27:43.313815 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.313793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.316346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.316325    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f385e1-serving-cert\""
Jul 27 16:27:43.325026 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.325004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:27:43.386244 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.386207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.386380 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.386257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.386380 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.386279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh54v\" (UniqueName: \"kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.486664 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.486630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.486828 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.486674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.486828 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.486717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh54v\" (UniqueName: \"kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.486941 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:27:43.486916    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-serving-cert: secret "maas-api-e2e-inf-b-f385e1-serving-cert" not found
Jul 27 16:27:43.487009 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:27:43.486996    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls podName:3b22ccbb-c574-467f-b27d-7984ac4b0b09 nodeName:}" failed. No retries permitted until 2026-07-27 16:27:43.986973552 +0000 UTC m=+2359.901030081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls") pod "maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" (UID: "3b22ccbb-c574-467f-b27d-7984ac4b0b09") : secret "maas-api-e2e-inf-b-f385e1-serving-cert" not found
Jul 27 16:27:43.487298 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.487278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.496054 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.496034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh54v\" (UniqueName: \"kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.723397 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.723366    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:27:43.989956 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.989875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:43.992284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:43.992255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") pod \"maas-api-e2e-inf-b-f385e1-849c464d76-66jsv\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:44.224644 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:44.224607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:44.366656 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:44.366629    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:27:44.368382 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:27:44.368354    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b22ccbb_c574_467f_b27d_7984ac4b0b09.slice/crio-cbd4b5f2c8feb710d8b9631e1e3b6627026c10b7ec836fa96de46da13026404d WatchSource:0}: Error finding container cbd4b5f2c8feb710d8b9631e1e3b6627026c10b7ec836fa96de46da13026404d: Status 404 returned error can't find the container with id cbd4b5f2c8feb710d8b9631e1e3b6627026c10b7ec836fa96de46da13026404d
Jul 27 16:27:44.747365 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:44.747330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" event={"ID":"3b22ccbb-c574-467f-b27d-7984ac4b0b09","Type":"ContainerStarted","Data":"cbd4b5f2c8feb710d8b9631e1e3b6627026c10b7ec836fa96de46da13026404d"}
Jul 27 16:27:45.753015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:45.752981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" event={"ID":"3b22ccbb-c574-467f-b27d-7984ac4b0b09","Type":"ContainerStarted","Data":"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"}
Jul 27 16:27:45.753407 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:45.753150    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:45.771655 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:45.771599    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" podStartSLOduration=2.294268996 podStartE2EDuration="2.77158631s" totalImagesPullingTime="477.317314ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:27:45.770690222 +0000 UTC m=+2361.684746770" watchObservedRunningTime="2026-07-27 16:27:45.77158631 +0000 UTC m=+2361.685642858"
Jul 27 16:27:49.216962 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:49.216931    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:27:51.764940 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:51.764903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:27:59.494903 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.494867    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:27:59.499554 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.499532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.502489 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.502471    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8cec93\"/\"kube-root-ca.crt\""
Jul 27 16:27:59.503670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.503647    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8cec93\"/\"default-dockercfg-xw5rj\""
Jul 27 16:27:59.503670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.503664    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8cec93\"/\"openshift-service-ca.crt\""
Jul 27 16:27:59.503817 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.503664    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8cec93\"/\"test-model-8cec93-kserve-self-signed-certs\""
Jul 27 16:27:59.506622 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.506600    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:27:59.516268 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.516388 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.516388 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.516574 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqqxt\" (UniqueName: \"kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.516574 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.516574 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.516568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.617808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618001 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618001 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618001 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618001 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618001 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.617991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqqxt\" (UniqueName: \"kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.618255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.618275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.618410 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.618392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.620220 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.620197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.620543 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.620525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.627253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.627228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqqxt\" (UniqueName: \"kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt\") pod \"test-model-8cec93-kserve-7fc96796cf-vfnfq\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") " pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.809657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.809571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:27:59.941075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:27:59.940994    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:27:59.943902 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:27:59.943877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65ddda31_593f_478a_99f8_0f5545482650.slice/crio-dcf3d6f6f62f4b988d945d1206cef4d81cf16826cd79f5cef0e0e6697f11cf40 WatchSource:0}: Error finding container dcf3d6f6f62f4b988d945d1206cef4d81cf16826cd79f5cef0e0e6697f11cf40: Status 404 returned error can't find the container with id dcf3d6f6f62f4b988d945d1206cef4d81cf16826cd79f5cef0e0e6697f11cf40
Jul 27 16:28:00.821832 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:00.821797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" event={"ID":"65ddda31-593f-478a-99f8-0f5545482650","Type":"ContainerStarted","Data":"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"}
Jul 27 16:28:00.821832 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:00.821835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" event={"ID":"65ddda31-593f-478a-99f8-0f5545482650","Type":"ContainerStarted","Data":"dcf3d6f6f62f4b988d945d1206cef4d81cf16826cd79f5cef0e0e6697f11cf40"}
Jul 27 16:28:00.839820 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:00.839771    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" podStartSLOduration=1.83975067 podStartE2EDuration="1.83975067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:28:00.839352627 +0000 UTC m=+2376.753409176" watchObservedRunningTime="2026-07-27 16:28:00.83975067 +0000 UTC m=+2376.753807219"
Jul 27 16:28:02.898094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:02.898060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:28:09.809759 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:09.809728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:28:09.810257 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:09.809802    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:28:09.822394 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:09.822370    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:28:09.875229 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:09.875200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:28:13.805214 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:13.805180    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:28:24.656381 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:24.656350    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:28:24.671366 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:24.671338    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:28:44.813237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.813201    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:28:44.817228 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.817209    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.821309 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.821275    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f385e1\"/\"kube-root-ca.crt\""
Jul 27 16:28:44.821463 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.821288    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f385e1\"/\"openshift-service-ca.crt\""
Jul 27 16:28:44.821463 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.821382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f385e1\"/\"test-model-f385e1-kserve-self-signed-certs\""
Jul 27 16:28:44.821463 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.821396    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f385e1\"/\"default-dockercfg-zcqb6\""
Jul 27 16:28:44.827253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.827230    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:28:44.919050 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.919252 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.919252 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.919252 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.919399 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:44.919399 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:44.919284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsbz\" (UniqueName: \"kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsbz\" (UniqueName: \"kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020735 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.020805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.020770    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.022600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.022578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.022700 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.022645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.029892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.029867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsbz\" (UniqueName: \"kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz\") pod \"test-model-f385e1-kserve-646b7658cc-nkzgl\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") " pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.127653 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.127578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:45.257944 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:45.257918    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:28:45.259727 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:28:45.259698    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24f3eee3_2c8d_41b2_ba90_953cf2453a58.slice/crio-9480dca9a16a548a236a54e360fe1308e1bc5eff520f41cabd57099b5e4ed792 WatchSource:0}: Error finding container 9480dca9a16a548a236a54e360fe1308e1bc5eff520f41cabd57099b5e4ed792: Status 404 returned error can't find the container with id 9480dca9a16a548a236a54e360fe1308e1bc5eff520f41cabd57099b5e4ed792
Jul 27 16:28:46.034927 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:46.034890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" event={"ID":"24f3eee3-2c8d-41b2-ba90-953cf2453a58","Type":"ContainerStarted","Data":"00e848e85b4a74fe19b0e26b17b86ca46927f5db799751ea4f11b8018d1ad06e"}
Jul 27 16:28:46.035412 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:46.034947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" event={"ID":"24f3eee3-2c8d-41b2-ba90-953cf2453a58","Type":"ContainerStarted","Data":"9480dca9a16a548a236a54e360fe1308e1bc5eff520f41cabd57099b5e4ed792"}
Jul 27 16:28:46.055486 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:46.055430    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" podStartSLOduration=2.05541293 podStartE2EDuration="2.05541293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:28:46.053635288 +0000 UTC m=+2421.967691840" watchObservedRunningTime="2026-07-27 16:28:46.05541293 +0000 UTC m=+2421.969469477"
Jul 27 16:28:47.608205 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:47.608166    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:28:55.127794 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:55.127758    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:55.128288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:55.127844    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:55.144259 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:55.144237    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:28:56.092481 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:28:56.092448    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:29:28.178114 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:28.178069    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:29:29.073220 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:29.073182    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:29:32.213323 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:32.213286    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:29:39.602165 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.602127    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:29:39.602971 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.602933    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" podUID="eea96160-5763-4fd3-8144-32a4fb11f464" containerName="maas-api" containerID="cri-o://306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce" gracePeriod=30
Jul 27 16:29:39.864809 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.864751    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:29:39.913574 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.913545    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") pod \"eea96160-5763-4fd3-8144-32a4fb11f464\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") "
Jul 27 16:29:39.913727 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.913600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzwwh\" (UniqueName: \"kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh\") pod \"eea96160-5763-4fd3-8144-32a4fb11f464\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") "
Jul 27 16:29:39.913727 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.913641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca\") pod \"eea96160-5763-4fd3-8144-32a4fb11f464\" (UID: \"eea96160-5763-4fd3-8144-32a4fb11f464\") "
Jul 27 16:29:39.914039 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.914017    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eea96160-5763-4fd3-8144-32a4fb11f464" (UID: "eea96160-5763-4fd3-8144-32a4fb11f464"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:29:39.915716 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.915685    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh" (OuterVolumeSpecName: "kube-api-access-mzwwh") pod "eea96160-5763-4fd3-8144-32a4fb11f464" (UID: "eea96160-5763-4fd3-8144-32a4fb11f464"). InnerVolumeSpecName "kube-api-access-mzwwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:29:39.915810 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:39.915718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eea96160-5763-4fd3-8144-32a4fb11f464" (UID: "eea96160-5763-4fd3-8144-32a4fb11f464"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:29:40.015246 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.015209    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eea96160-5763-4fd3-8144-32a4fb11f464-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:40.015246 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.015235    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzwwh\" (UniqueName: \"kubernetes.io/projected/eea96160-5763-4fd3-8144-32a4fb11f464-kube-api-access-mzwwh\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:40.015246 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.015245    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea96160-5763-4fd3-8144-32a4fb11f464-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:40.288637 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.288598    2576 generic.go:356] "Generic (PLEG): container finished" podID="eea96160-5763-4fd3-8144-32a4fb11f464" containerID="306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce" exitCode=0
Jul 27 16:29:40.288825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.288649    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"
Jul 27 16:29:40.288825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.288653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" event={"ID":"eea96160-5763-4fd3-8144-32a4fb11f464","Type":"ContainerDied","Data":"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"}
Jul 27 16:29:40.288825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.288767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk" event={"ID":"eea96160-5763-4fd3-8144-32a4fb11f464","Type":"ContainerDied","Data":"c811d518f1830d03c9ba32479de4d7771a41d51f9b7453719d1575772682a8d5"}
Jul 27 16:29:40.288825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.288789    2576 scope.go:117] "RemoveContainer" containerID="306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"
Jul 27 16:29:40.298094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.298076    2576 scope.go:117] "RemoveContainer" containerID="306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"
Jul 27 16:29:40.298425 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:29:40.298404    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce\": container with ID starting with 306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce not found: ID does not exist" containerID="306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"
Jul 27 16:29:40.298518 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.298437    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce"} err="failed to get container status \"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce\": rpc error: code = NotFound desc = could not find container \"306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce\": container with ID starting with 306e3467db23b749578a65e7b4cf1d7069d428015a0a1c28049464ace9ad08ce not found: ID does not exist"
Jul 27 16:29:40.313669 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.313633    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:29:40.318020 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.317998    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8cec93-589bdb64cb-qcqtk"]
Jul 27 16:29:40.653393 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:40.653317    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea96160-5763-4fd3-8144-32a4fb11f464" path="/var/lib/kubelet/pods/eea96160-5763-4fd3-8144-32a4fb11f464/volumes"
Jul 27 16:29:43.196695 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.196653    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" podUID="65ddda31-593f-478a-99f8-0f5545482650" containerName="main" containerID="cri-o://d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0" gracePeriod=45
Jul 27 16:29:43.447985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.447932    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:29:43.543781 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.543756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.543947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.543791    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.543947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.543822    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqqxt\" (UniqueName: \"kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.543947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.543847    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.544144 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.543967    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.544144 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544054    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm\") pod \"65ddda31-593f-478a-99f8-0f5545482650\" (UID: \"65ddda31-593f-478a-99f8-0f5545482650\") "
Jul 27 16:29:43.544144 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544056    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache" (OuterVolumeSpecName: "model-cache") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:43.544144 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544065    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:43.544320 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544254    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home" (OuterVolumeSpecName: "home") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:43.544431 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544415    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:43.544483 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544434    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:43.544483 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.544442    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:43.546244 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.546221    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm" (OuterVolumeSpecName: "dshm") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:43.546411 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.546392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:29:43.546471 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.546415    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt" (OuterVolumeSpecName: "kube-api-access-gqqxt") pod "65ddda31-593f-478a-99f8-0f5545482650" (UID: "65ddda31-593f-478a-99f8-0f5545482650"). InnerVolumeSpecName "kube-api-access-gqqxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:29:43.645113 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.645082    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ddda31-593f-478a-99f8-0f5545482650-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:43.645237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.645119    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ddda31-593f-478a-99f8-0f5545482650-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:43.645237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:43.645130    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqqxt\" (UniqueName: \"kubernetes.io/projected/65ddda31-593f-478a-99f8-0f5545482650-kube-api-access-gqqxt\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.091980 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.091914    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" podUID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" containerName="main" containerID="cri-o://00e848e85b4a74fe19b0e26b17b86ca46927f5db799751ea4f11b8018d1ad06e" gracePeriod=45
Jul 27 16:29:44.308466 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.308435    2576 generic.go:356] "Generic (PLEG): container finished" podID="65ddda31-593f-478a-99f8-0f5545482650" containerID="d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0" exitCode=0
Jul 27 16:29:44.308892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.308478    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"
Jul 27 16:29:44.308892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.308520    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" event={"ID":"65ddda31-593f-478a-99f8-0f5545482650","Type":"ContainerDied","Data":"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"}
Jul 27 16:29:44.308892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.308568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq" event={"ID":"65ddda31-593f-478a-99f8-0f5545482650","Type":"ContainerDied","Data":"dcf3d6f6f62f4b988d945d1206cef4d81cf16826cd79f5cef0e0e6697f11cf40"}
Jul 27 16:29:44.308892 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.308588    2576 scope.go:117] "RemoveContainer" containerID="d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"
Jul 27 16:29:44.310221 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.310198    2576 generic.go:356] "Generic (PLEG): container finished" podID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" containerID="00e848e85b4a74fe19b0e26b17b86ca46927f5db799751ea4f11b8018d1ad06e" exitCode=0
Jul 27 16:29:44.310359 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.310273    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" event={"ID":"24f3eee3-2c8d-41b2-ba90-953cf2453a58","Type":"ContainerDied","Data":"00e848e85b4a74fe19b0e26b17b86ca46927f5db799751ea4f11b8018d1ad06e"}
Jul 27 16:29:44.317959 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.317944    2576 scope.go:117] "RemoveContainer" containerID="d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"
Jul 27 16:29:44.318251 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:29:44.318234    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0\": container with ID starting with d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0 not found: ID does not exist" containerID="d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"
Jul 27 16:29:44.318316 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.318258    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0"} err="failed to get container status \"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0\": rpc error: code = NotFound desc = could not find container \"d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0\": container with ID starting with d4d95ea510aef4e634ebd7ec827fed373465c173894d07be75a0647fc26c2cc0 not found: ID does not exist"
Jul 27 16:29:44.334010 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.333986    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:29:44.348350 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.348300    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:29:44.350479 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.350451    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8cec93/test-model-8cec93-kserve-7fc96796cf-vfnfq"]
Jul 27 16:29:44.452901 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.452876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453016 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.452913    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453016 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.452947    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453177 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453058    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwsbz\" (UniqueName: \"kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453177 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453287 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home" (OuterVolumeSpecName: "home") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:44.453287 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453199    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:44.453287 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453208    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache\") pod \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\" (UID: \"24f3eee3-2c8d-41b2-ba90-953cf2453a58\") "
Jul 27 16:29:44.453419 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453373    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache" (OuterVolumeSpecName: "model-cache") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:44.453543 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453519    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.453622 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453544    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.453622 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.453553    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.455117 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.455079    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:29:44.455211 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.455154    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm" (OuterVolumeSpecName: "dshm") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:29:44.455211 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.455176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz" (OuterVolumeSpecName: "kube-api-access-bwsbz") pod "24f3eee3-2c8d-41b2-ba90-953cf2453a58" (UID: "24f3eee3-2c8d-41b2-ba90-953cf2453a58"). InnerVolumeSpecName "kube-api-access-bwsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:29:44.554217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.554194    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwsbz\" (UniqueName: \"kubernetes.io/projected/24f3eee3-2c8d-41b2-ba90-953cf2453a58-kube-api-access-bwsbz\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.554217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.554216    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24f3eee3-2c8d-41b2-ba90-953cf2453a58-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.554217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.554225    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24f3eee3-2c8d-41b2-ba90-953cf2453a58-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:44.653881 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.653822    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ddda31-593f-478a-99f8-0f5545482650" path="/var/lib/kubelet/pods/65ddda31-593f-478a-99f8-0f5545482650/volumes"
Jul 27 16:29:44.894745 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:44.894712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:29:45.317384 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:45.317353    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"
Jul 27 16:29:45.317384 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:45.317377    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl" event={"ID":"24f3eee3-2c8d-41b2-ba90-953cf2453a58","Type":"ContainerDied","Data":"9480dca9a16a548a236a54e360fe1308e1bc5eff520f41cabd57099b5e4ed792"}
Jul 27 16:29:45.317929 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:45.317421    2576 scope.go:117] "RemoveContainer" containerID="00e848e85b4a74fe19b0e26b17b86ca46927f5db799751ea4f11b8018d1ad06e"
Jul 27 16:29:45.339308 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:45.339276    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:29:45.346832 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:45.346803    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f385e1/test-model-f385e1-kserve-646b7658cc-nkzgl"]
Jul 27 16:29:46.651690 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:46.651659    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" path="/var/lib/kubelet/pods/24f3eee3-2c8d-41b2-ba90-953cf2453a58/volumes"
Jul 27 16:29:51.683864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.683830    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"]
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684171    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ddda31-593f-478a-99f8-0f5545482650" containerName="main"
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684182    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ddda31-593f-478a-99f8-0f5545482650" containerName="main"
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684194    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea96160-5763-4fd3-8144-32a4fb11f464" containerName="maas-api"
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684200    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea96160-5763-4fd3-8144-32a4fb11f464" containerName="maas-api"
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684207    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" containerName="main"
Jul 27 16:29:51.684258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684212    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" containerName="main"
Jul 27 16:29:51.684507 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684274    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ddda31-593f-478a-99f8-0f5545482650" containerName="main"
Jul 27 16:29:51.684507 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684283    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea96160-5763-4fd3-8144-32a4fb11f464" containerName="maas-api"
Jul 27 16:29:51.684507 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.684291    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f3eee3-2c8d-41b2-ba90-953cf2453a58" containerName="main"
Jul 27 16:29:51.688720 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.688701    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:51.691166 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.691146    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:29:51.695290 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.695263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"]
Jul 27 16:29:51.821282 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.821255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx\" (UID: \"cdf85409-fd3b-4875-ba84-402b3a43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:51.922371 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.922337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx\" (UID: \"cdf85409-fd3b-4875-ba84-402b3a43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:51.922914 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:51.922895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx\" (UID: \"cdf85409-fd3b-4875-ba84-402b3a43e4b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:52.000461 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:52.000431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:52.132862 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:52.132836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"]
Jul 27 16:29:52.135653 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:29:52.135622    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf85409_fd3b_4875_ba84_402b3a43e4b3.slice/crio-4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5 WatchSource:0}: Error finding container 4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5: Status 404 returned error can't find the container with id 4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5
Jul 27 16:29:52.137976 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:52.137956    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:29:52.350817 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:52.350738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx" event={"ID":"cdf85409-fd3b-4875-ba84-402b3a43e4b3","Type":"ContainerStarted","Data":"4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5"}
Jul 27 16:29:56.373313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:56.373277    2576 generic.go:356] "Generic (PLEG): container finished" podID="cdf85409-fd3b-4875-ba84-402b3a43e4b3" containerID="921ac8985860f612cbfb17961bf84b3548de718a25ef532860e51dc978c0e72d" exitCode=0
Jul 27 16:29:56.373698 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:56.373321    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx" event={"ID":"cdf85409-fd3b-4875-ba84-402b3a43e4b3","Type":"ContainerDied","Data":"921ac8985860f612cbfb17961bf84b3548de718a25ef532860e51dc978c0e72d"}
Jul 27 16:29:57.504674 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:57.504645    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:57.573861 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:57.573832    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca\") pod \"cdf85409-fd3b-4875-ba84-402b3a43e4b3\" (UID: \"cdf85409-fd3b-4875-ba84-402b3a43e4b3\") "
Jul 27 16:29:57.574189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:57.574170    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cdf85409-fd3b-4875-ba84-402b3a43e4b3" (UID: "cdf85409-fd3b-4875-ba84-402b3a43e4b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:29:57.675364 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:57.675297    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdf85409-fd3b-4875-ba84-402b3a43e4b3-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:29:58.383572 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:58.383540    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"
Jul 27 16:29:58.383572 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:58.383551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx" event={"ID":"cdf85409-fd3b-4875-ba84-402b3a43e4b3","Type":"ContainerDied","Data":"4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5"}
Jul 27 16:29:58.383805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:29:58.383594    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5b9edb50a62f81dbe1a38372895d5999b8757dba321964565f981f073ccda5"
Jul 27 16:30:00.142807 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.142773    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"]
Jul 27 16:30:00.144988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.143153    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf85409-fd3b-4875-ba84-402b3a43e4b3" containerName="revoke-keys"
Jul 27 16:30:00.144988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.143167    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf85409-fd3b-4875-ba84-402b3a43e4b3" containerName="revoke-keys"
Jul 27 16:30:00.144988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.143229    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf85409-fd3b-4875-ba84-402b3a43e4b3" containerName="revoke-keys"
Jul 27 16:30:00.145829 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.145813    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"
Jul 27 16:30:00.148409 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.148387    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:30:00.155178 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.155144    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"]
Jul 27 16:30:00.156725 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.156706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"
Jul 27 16:30:00.294878 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.294855    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"]
Jul 27 16:30:00.296902 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:30:00.296877    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa95cee2_89e5_415f_899f_612dba8d1a98.slice/crio-b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0 WatchSource:0}: Error finding container b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0: Status 404 returned error can't find the container with id b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0
Jul 27 16:30:00.394754 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:00.394681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm" event={"ID":"aa95cee2-89e5-415f-899f-612dba8d1a98","Type":"ContainerStarted","Data":"b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0"}
Jul 27 16:30:01.401066 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:01.400974    2576 generic.go:356] "Generic (PLEG): container finished" podID="aa95cee2-89e5-415f-899f-612dba8d1a98" containerID="6a4c258f1b8cd5c3bc25411cb68d166d41519d1c67d4c0d9a99fbff667f923da" exitCode=0
Jul 27 16:30:01.401066 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:01.401058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm" event={"ID":"aa95cee2-89e5-415f-899f-612dba8d1a98","Type":"ContainerDied","Data":"6a4c258f1b8cd5c3bc25411cb68d166d41519d1c67d4c0d9a99fbff667f923da"}
Jul 27 16:30:02.113452 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.113392    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:30:02.113769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.113707    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" podUID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" containerName="maas-api" containerID="cri-o://59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb" gracePeriod=30
Jul 27 16:30:02.362734 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.362709    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:30:02.408423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.408340    2576 generic.go:356] "Generic (PLEG): container finished" podID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" containerID="59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb" exitCode=0
Jul 27 16:30:02.408423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.408406    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"
Jul 27 16:30:02.408858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.408429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" event={"ID":"3b22ccbb-c574-467f-b27d-7984ac4b0b09","Type":"ContainerDied","Data":"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"}
Jul 27 16:30:02.408858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.408479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv" event={"ID":"3b22ccbb-c574-467f-b27d-7984ac4b0b09","Type":"ContainerDied","Data":"cbd4b5f2c8feb710d8b9631e1e3b6627026c10b7ec836fa96de46da13026404d"}
Jul 27 16:30:02.408858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.408501    2576 scope.go:117] "RemoveContainer" containerID="59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"
Jul 27 16:30:02.417130 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.417091    2576 scope.go:117] "RemoveContainer" containerID="59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"
Jul 27 16:30:02.417394 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:30:02.417372    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb\": container with ID starting with 59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb not found: ID does not exist" containerID="59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"
Jul 27 16:30:02.417458 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.417403    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb"} err="failed to get container status \"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb\": rpc error: code = NotFound desc = could not find container \"59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb\": container with ID starting with 59f34455a4bb9421e443b2e6f06c3c09efbddbc45eff7357bddf29959fca58bb not found: ID does not exist"
Jul 27 16:30:02.520580 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.520546    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") pod \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") "
Jul 27 16:30:02.520746 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.520630    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca\") pod \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") "
Jul 27 16:30:02.520746 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.520683    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh54v\" (UniqueName: \"kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v\") pod \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\" (UID: \"3b22ccbb-c574-467f-b27d-7984ac4b0b09\") "
Jul 27 16:30:02.521029 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.521000    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b22ccbb-c574-467f-b27d-7984ac4b0b09" (UID: "3b22ccbb-c574-467f-b27d-7984ac4b0b09"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:30:02.522695 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.522672    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b22ccbb-c574-467f-b27d-7984ac4b0b09" (UID: "3b22ccbb-c574-467f-b27d-7984ac4b0b09"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:30:02.524097 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.524070    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v" (OuterVolumeSpecName: "kube-api-access-qh54v") pod "3b22ccbb-c574-467f-b27d-7984ac4b0b09" (UID: "3b22ccbb-c574-467f-b27d-7984ac4b0b09"). InnerVolumeSpecName "kube-api-access-qh54v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:30:02.557836 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.557813    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"
Jul 27 16:30:02.622176 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.622141    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh54v\" (UniqueName: \"kubernetes.io/projected/3b22ccbb-c574-467f-b27d-7984ac4b0b09-kube-api-access-qh54v\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:02.622176 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.622171    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b22ccbb-c574-467f-b27d-7984ac4b0b09-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:02.622176 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.622180    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b22ccbb-c574-467f-b27d-7984ac4b0b09-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:02.725609 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.725580    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:30:02.730261 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:02.730234    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f385e1-849c464d76-66jsv"]
Jul 27 16:30:03.413571 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:03.413493    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm"
Jul 27 16:30:03.413965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:03.413492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752830-w4vzm" event={"ID":"aa95cee2-89e5-415f-899f-612dba8d1a98","Type":"ContainerDied","Data":"b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0"}
Jul 27 16:30:03.413965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:03.413640    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b62568b0b63be15de41f51c6751bca0ba353c5c7a2aa1659a49134efad0489d0"
Jul 27 16:30:04.652466 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:04.652431    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" path="/var/lib/kubelet/pods/3b22ccbb-c574-467f-b27d-7984ac4b0b09/volumes"
Jul 27 16:30:04.907040 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:04.906961    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:30:06.756967 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:06.756931    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"]
Jul 27 16:30:06.762771 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:06.762741    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f385e1-86xpx"]
Jul 27 16:30:08.652272 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:08.652238    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf85409-fd3b-4875-ba84-402b3a43e4b3" path="/var/lib/kubelet/pods/cdf85409-fd3b-4875-ba84-402b3a43e4b3/volumes"
Jul 27 16:30:18.261430 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.261230    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:18.262321 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" containerName="maas-api"
Jul 27 16:30:18.262395 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262325    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" containerName="maas-api"
Jul 27 16:30:18.262395 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262340    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa95cee2-89e5-415f-899f-612dba8d1a98" containerName="cleanup"
Jul 27 16:30:18.262395 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262350    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa95cee2-89e5-415f-899f-612dba8d1a98" containerName="cleanup"
Jul 27 16:30:18.262488 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262448    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b22ccbb-c574-467f-b27d-7984ac4b0b09" containerName="maas-api"
Jul 27 16:30:18.262488 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.262460    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa95cee2-89e5-415f-899f-612dba8d1a98" containerName="cleanup"
Jul 27 16:30:18.266868 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.266847    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.269772 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.269749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:30:18.270768 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.270748    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:30:18.270871 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.270755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:30:18.271736 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.271719    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d2ac7b-serving-cert\""
Jul 27 16:30:18.277317 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.277293    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:18.357932 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.357897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.358094 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.357950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd694\" (UniqueName: \"kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.358173 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.358084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.459295 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.459260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.459469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.459311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.459469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.459345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd694\" (UniqueName: \"kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.460088 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.460062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.461770 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.461746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.469368 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.469349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd694\" (UniqueName: \"kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694\") pod \"maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.580277 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.580198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:18.715967 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:18.715929    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:18.718260 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:30:18.718228    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3824b92e_5228_4a59_93b4_0178964a988e.slice/crio-09f2e88c00ca112467a2da02107760a3f5cf17ff5b819d519a6790162165bea3 WatchSource:0}: Error finding container 09f2e88c00ca112467a2da02107760a3f5cf17ff5b819d519a6790162165bea3: Status 404 returned error can't find the container with id 09f2e88c00ca112467a2da02107760a3f5cf17ff5b819d519a6790162165bea3
Jul 27 16:30:19.496851 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:19.496816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" event={"ID":"3824b92e-5228-4a59-93b4-0178964a988e","Type":"ContainerStarted","Data":"f38c261ee658e03c4142657a4527dda57b6c96cb63a6de3db450f1bebcd591f5"}
Jul 27 16:30:19.497253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:19.496865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" event={"ID":"3824b92e-5228-4a59-93b4-0178964a988e","Type":"ContainerStarted","Data":"09f2e88c00ca112467a2da02107760a3f5cf17ff5b819d519a6790162165bea3"}
Jul 27 16:30:19.497253 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:19.496946    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:19.513647 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:19.513562    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" podStartSLOduration=0.975771065 podStartE2EDuration="1.513549028s" totalImagesPullingTime="537.777963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:30:19.512850155 +0000 UTC m=+2515.426906722" watchObservedRunningTime="2026-07-27 16:30:19.513549028 +0000 UTC m=+2515.427605575"
Jul 27 16:30:20.896652 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:20.896616    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:30:25.507244 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:25.507215    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:26.286754 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.286714    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:30:26.297513 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.297483    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.300037 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.300016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-478f20-serving-cert\""
Jul 27 16:30:26.308252 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.308227    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:30:26.333395 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.333366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t7vt\" (UniqueName: \"kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.333553 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.333441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.333553 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.333473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.434480 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.434450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t7vt\" (UniqueName: \"kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.434637 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.434511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.434637 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.434537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.434804 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:30:26.434639    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-serving-cert: secret "maas-api-e2e-api-b-478f20-serving-cert" not found
Jul 27 16:30:26.434804 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:30:26.434718    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls podName:a869756b-781c-46f9-8c06-63d42f6c2765 nodeName:}" failed. No retries permitted until 2026-07-27 16:30:26.93469612 +0000 UTC m=+2522.848752661 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls") pod "maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" (UID: "a869756b-781c-46f9-8c06-63d42f6c2765") : secret "maas-api-e2e-api-b-478f20-serving-cert" not found
Jul 27 16:30:26.435303 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.435280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.443339 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.443315    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t7vt\" (UniqueName: \"kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.939415 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.939378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:26.941899 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:26.941867    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") pod \"maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:27.209606 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:27.209528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:27.357305 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:27.357280    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:30:27.359329 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:30:27.359303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda869756b_781c_46f9_8c06_63d42f6c2765.slice/crio-fabeec6b1c34e9c9237c8ac946baacc56b94efa4feac408efbf737453576bc06 WatchSource:0}: Error finding container fabeec6b1c34e9c9237c8ac946baacc56b94efa4feac408efbf737453576bc06: Status 404 returned error can't find the container with id fabeec6b1c34e9c9237c8ac946baacc56b94efa4feac408efbf737453576bc06
Jul 27 16:30:27.540090 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:27.540006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" event={"ID":"a869756b-781c-46f9-8c06-63d42f6c2765","Type":"ContainerStarted","Data":"fabeec6b1c34e9c9237c8ac946baacc56b94efa4feac408efbf737453576bc06"}
Jul 27 16:30:28.547789 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:28.547753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" event={"ID":"a869756b-781c-46f9-8c06-63d42f6c2765","Type":"ContainerStarted","Data":"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"}
Jul 27 16:30:28.548213 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:28.547813    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:28.568749 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:28.568700    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" podStartSLOduration=2.16639584 podStartE2EDuration="2.568681045s" totalImagesPullingTime="402.285205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:30:28.568263895 +0000 UTC m=+2524.482320469" watchObservedRunningTime="2026-07-27 16:30:28.568681045 +0000 UTC m=+2524.482737593"
Jul 27 16:30:33.310075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:33.310038    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:30:34.557851 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:34.557817    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:30:38.821598 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:38.821568    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"]
Jul 27 16:30:38.824814 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:38.824793    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:38.827373 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:38.827352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:30:38.833237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:38.833209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"]
Jul 27 16:30:38.946685 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:38.946652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g\" (UID: \"005f9f61-26f7-4214-8c88-df8e93444fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:39.047385 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.047347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g\" (UID: \"005f9f61-26f7-4214-8c88-df8e93444fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:39.047954 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.047935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g\" (UID: \"005f9f61-26f7-4214-8c88-df8e93444fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:39.136064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.135971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:39.269469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.269428    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"]
Jul 27 16:30:39.271510 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:30:39.271486    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005f9f61_26f7_4214_8c88_df8e93444fab.slice/crio-23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad WatchSource:0}: Error finding container 23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad: Status 404 returned error can't find the container with id 23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad
Jul 27 16:30:39.598176 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.598141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g" event={"ID":"005f9f61-26f7-4214-8c88-df8e93444fab","Type":"ContainerStarted","Data":"e98481965987494c9b482d13148dcbe43f710041e6aaf08c31e191ccd386be33"}
Jul 27 16:30:39.598353 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.598195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g" event={"ID":"005f9f61-26f7-4214-8c88-df8e93444fab","Type":"ContainerStarted","Data":"23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad"}
Jul 27 16:30:39.614508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:39.614460    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g" podStartSLOduration=1.6144461620000001 podStartE2EDuration="1.614446162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:30:39.613746771 +0000 UTC m=+2535.527803320" watchObservedRunningTime="2026-07-27 16:30:39.614446162 +0000 UTC m=+2535.528502709"
Jul 27 16:30:40.604171 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:40.604141    2576 generic.go:356] "Generic (PLEG): container finished" podID="005f9f61-26f7-4214-8c88-df8e93444fab" containerID="e98481965987494c9b482d13148dcbe43f710041e6aaf08c31e191ccd386be33" exitCode=0
Jul 27 16:30:40.604529 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:40.604201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g" event={"ID":"005f9f61-26f7-4214-8c88-df8e93444fab","Type":"ContainerDied","Data":"e98481965987494c9b482d13148dcbe43f710041e6aaf08c31e191ccd386be33"}
Jul 27 16:30:41.738869 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:41.738846    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:41.873098 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:41.873023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca\") pod \"005f9f61-26f7-4214-8c88-df8e93444fab\" (UID: \"005f9f61-26f7-4214-8c88-df8e93444fab\") "
Jul 27 16:30:41.873421 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:41.873396    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "005f9f61-26f7-4214-8c88-df8e93444fab" (UID: "005f9f61-26f7-4214-8c88-df8e93444fab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:30:41.974212 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:41.974166    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005f9f61-26f7-4214-8c88-df8e93444fab-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:42.613974 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:42.613947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"
Jul 27 16:30:42.613974 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:42.613964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g" event={"ID":"005f9f61-26f7-4214-8c88-df8e93444fab","Type":"ContainerDied","Data":"23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad"}
Jul 27 16:30:42.614206 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:42.614005    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23c1beba596fea9c74c3f9f53a39c31a6fc0018f687b91a8c5fc3acdcc03f5ad"
Jul 27 16:30:54.495690 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.495657    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:54.496260 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.495898    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" podUID="3824b92e-5228-4a59-93b4-0178964a988e" containerName="maas-api" containerID="cri-o://f38c261ee658e03c4142657a4527dda57b6c96cb63a6de3db450f1bebcd591f5" gracePeriod=30
Jul 27 16:30:54.669246 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.669219    2576 generic.go:356] "Generic (PLEG): container finished" podID="3824b92e-5228-4a59-93b4-0178964a988e" containerID="f38c261ee658e03c4142657a4527dda57b6c96cb63a6de3db450f1bebcd591f5" exitCode=0
Jul 27 16:30:54.669348 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.669277    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" event={"ID":"3824b92e-5228-4a59-93b4-0178964a988e","Type":"ContainerDied","Data":"f38c261ee658e03c4142657a4527dda57b6c96cb63a6de3db450f1bebcd591f5"}
Jul 27 16:30:54.749495 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.749441    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:54.891042 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.891010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd694\" (UniqueName: \"kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694\") pod \"3824b92e-5228-4a59-93b4-0178964a988e\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") "
Jul 27 16:30:54.891207 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.891083    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca\") pod \"3824b92e-5228-4a59-93b4-0178964a988e\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") "
Jul 27 16:30:54.891207 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.891127    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls\") pod \"3824b92e-5228-4a59-93b4-0178964a988e\" (UID: \"3824b92e-5228-4a59-93b4-0178964a988e\") "
Jul 27 16:30:54.891498 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.891461    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3824b92e-5228-4a59-93b4-0178964a988e" (UID: "3824b92e-5228-4a59-93b4-0178964a988e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:30:54.893035 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.893011    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694" (OuterVolumeSpecName: "kube-api-access-bd694") pod "3824b92e-5228-4a59-93b4-0178964a988e" (UID: "3824b92e-5228-4a59-93b4-0178964a988e"). InnerVolumeSpecName "kube-api-access-bd694". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:30:54.893234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.893210    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3824b92e-5228-4a59-93b4-0178964a988e" (UID: "3824b92e-5228-4a59-93b4-0178964a988e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:30:54.991912 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.991884    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd694\" (UniqueName: \"kubernetes.io/projected/3824b92e-5228-4a59-93b4-0178964a988e-kube-api-access-bd694\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:54.991912 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.991907    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3824b92e-5228-4a59-93b4-0178964a988e-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:54.991912 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:54.991918    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3824b92e-5228-4a59-93b4-0178964a988e-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:30:55.675373 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:55.675344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86" event={"ID":"3824b92e-5228-4a59-93b4-0178964a988e","Type":"ContainerDied","Data":"09f2e88c00ca112467a2da02107760a3f5cf17ff5b819d519a6790162165bea3"}
Jul 27 16:30:55.675373 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:55.675370    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"
Jul 27 16:30:55.675831 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:55.675395    2576 scope.go:117] "RemoveContainer" containerID="f38c261ee658e03c4142657a4527dda57b6c96cb63a6de3db450f1bebcd591f5"
Jul 27 16:30:55.699396 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:55.699371    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:55.705072 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:55.705049    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d2ac7b-78d979559f-6jh86"]
Jul 27 16:30:56.652300 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:56.652258    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3824b92e-5228-4a59-93b4-0178964a988e" path="/var/lib/kubelet/pods/3824b92e-5228-4a59-93b4-0178964a988e/volumes"
Jul 27 16:30:56.693780 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:56.693746    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:30:58.902331 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:58.902294    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"]
Jul 27 16:30:58.908167 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:30:58.908140    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2ac7b-gts8g"]
Jul 27 16:31:00.652344 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:00.652311    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005f9f61-26f7-4214-8c88-df8e93444fab" path="/var/lib/kubelet/pods/005f9f61-26f7-4214-8c88-df8e93444fab/volumes"
Jul 27 16:31:06.149617 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.149582    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"]
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.149908    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3824b92e-5228-4a59-93b4-0178964a988e" containerName="maas-api"
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.149918    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3824b92e-5228-4a59-93b4-0178964a988e" containerName="maas-api"
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.149944    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005f9f61-26f7-4214-8c88-df8e93444fab" containerName="revoke-keys"
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.149950    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="005f9f61-26f7-4214-8c88-df8e93444fab" containerName="revoke-keys"
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.150001    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="005f9f61-26f7-4214-8c88-df8e93444fab" containerName="revoke-keys"
Jul 27 16:31:06.150027 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.150011    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3824b92e-5228-4a59-93b4-0178964a988e" containerName="maas-api"
Jul 27 16:31:06.152923 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.152903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:06.155329 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.155310    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:31:06.166219 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.166185    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"]
Jul 27 16:31:06.185587 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.185558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-478f20-vdflg\" (UID: \"8a176e76-2a86-4a6c-9c30-61205bbe42ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:06.286709 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.286678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-478f20-vdflg\" (UID: \"8a176e76-2a86-4a6c-9c30-61205bbe42ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:06.287415 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.287393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-478f20-vdflg\" (UID: \"8a176e76-2a86-4a6c-9c30-61205bbe42ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:06.462866 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.462839    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:06.613923 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:31:06.613893    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a176e76_2a86_4a6c_9c30_61205bbe42ba.slice/crio-8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056 WatchSource:0}: Error finding container 8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056: Status 404 returned error can't find the container with id 8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056
Jul 27 16:31:06.617170 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.617147    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"]
Jul 27 16:31:06.725508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:06.725438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg" event={"ID":"8a176e76-2a86-4a6c-9c30-61205bbe42ba","Type":"ContainerStarted","Data":"8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056"}
Jul 27 16:31:07.731245 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:07.731207    2576 generic.go:356] "Generic (PLEG): container finished" podID="8a176e76-2a86-4a6c-9c30-61205bbe42ba" containerID="3bae6eb7b6f590536e7584bbce5e68b4688da5ca4b897de24b9acadef9f1024d" exitCode=0
Jul 27 16:31:07.731665 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:07.731295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg" event={"ID":"8a176e76-2a86-4a6c-9c30-61205bbe42ba","Type":"ContainerDied","Data":"3bae6eb7b6f590536e7584bbce5e68b4688da5ca4b897de24b9acadef9f1024d"}
Jul 27 16:31:08.875498 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:08.875471    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:08.911061 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:08.911025    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca\") pod \"8a176e76-2a86-4a6c-9c30-61205bbe42ba\" (UID: \"8a176e76-2a86-4a6c-9c30-61205bbe42ba\") "
Jul 27 16:31:08.911489 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:08.911462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a176e76-2a86-4a6c-9c30-61205bbe42ba" (UID: "8a176e76-2a86-4a6c-9c30-61205bbe42ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:31:09.012479 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:09.012398    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a176e76-2a86-4a6c-9c30-61205bbe42ba-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:31:09.742374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:09.742336    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg" event={"ID":"8a176e76-2a86-4a6c-9c30-61205bbe42ba","Type":"ContainerDied","Data":"8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056"}
Jul 27 16:31:09.742540 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:09.742387    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c0c9acfae58ebc3de899101813cd8ddfd99b8730bce3575fc9230e2e182d056"
Jul 27 16:31:09.742540 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:09.742353    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"
Jul 27 16:31:21.742024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:21.741991    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:31:21.742884 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:21.742840    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" podUID="a869756b-781c-46f9-8c06-63d42f6c2765" containerName="maas-api" containerID="cri-o://1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30" gracePeriod=30
Jul 27 16:31:21.994836 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:21.994779    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:31:22.129712 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.129679    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") pod \"a869756b-781c-46f9-8c06-63d42f6c2765\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") "
Jul 27 16:31:22.129862 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.129774    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca\") pod \"a869756b-781c-46f9-8c06-63d42f6c2765\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") "
Jul 27 16:31:22.129862 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.129814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t7vt\" (UniqueName: \"kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt\") pod \"a869756b-781c-46f9-8c06-63d42f6c2765\" (UID: \"a869756b-781c-46f9-8c06-63d42f6c2765\") "
Jul 27 16:31:22.130165 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.130137    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a869756b-781c-46f9-8c06-63d42f6c2765" (UID: "a869756b-781c-46f9-8c06-63d42f6c2765"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:31:22.131686 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.131661    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a869756b-781c-46f9-8c06-63d42f6c2765" (UID: "a869756b-781c-46f9-8c06-63d42f6c2765"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:31:22.131803 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.131778    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt" (OuterVolumeSpecName: "kube-api-access-9t7vt") pod "a869756b-781c-46f9-8c06-63d42f6c2765" (UID: "a869756b-781c-46f9-8c06-63d42f6c2765"). InnerVolumeSpecName "kube-api-access-9t7vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:31:22.231114 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.231075    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a869756b-781c-46f9-8c06-63d42f6c2765-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:31:22.231114 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.231109    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t7vt\" (UniqueName: \"kubernetes.io/projected/a869756b-781c-46f9-8c06-63d42f6c2765-kube-api-access-9t7vt\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:31:22.231258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.231122    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a869756b-781c-46f9-8c06-63d42f6c2765-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:31:22.806356 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.806323    2576 generic.go:356] "Generic (PLEG): container finished" podID="a869756b-781c-46f9-8c06-63d42f6c2765" containerID="1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30" exitCode=0
Jul 27 16:31:22.806715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.806380    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"
Jul 27 16:31:22.806715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.806410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" event={"ID":"a869756b-781c-46f9-8c06-63d42f6c2765","Type":"ContainerDied","Data":"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"}
Jul 27 16:31:22.806715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.806468    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh" event={"ID":"a869756b-781c-46f9-8c06-63d42f6c2765","Type":"ContainerDied","Data":"fabeec6b1c34e9c9237c8ac946baacc56b94efa4feac408efbf737453576bc06"}
Jul 27 16:31:22.806715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.806497    2576 scope.go:117] "RemoveContainer" containerID="1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"
Jul 27 16:31:22.814282 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.814267    2576 scope.go:117] "RemoveContainer" containerID="1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"
Jul 27 16:31:22.814520 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:31:22.814502    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30\": container with ID starting with 1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30 not found: ID does not exist" containerID="1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"
Jul 27 16:31:22.814578 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.814527    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30"} err="failed to get container status \"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30\": rpc error: code = NotFound desc = could not find container \"1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30\": container with ID starting with 1b96c1514cdfe19d6df76072531a067a994a82e7115f50603562f4cb5c780a30 not found: ID does not exist"
Jul 27 16:31:22.824691 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.824670    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:31:22.830661 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:22.830642    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-478f20-bbfbb6c94-2fflh"]
Jul 27 16:31:24.652418 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:24.652383    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a869756b-781c-46f9-8c06-63d42f6c2765" path="/var/lib/kubelet/pods/a869756b-781c-46f9-8c06-63d42f6c2765/volumes"
Jul 27 16:31:26.229025 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:26.228992    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"]
Jul 27 16:31:26.233000 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:26.232977    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-478f20-vdflg"]
Jul 27 16:31:26.501931 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:26.501849    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:31:26.652480 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:26.652449    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a176e76-2a86-4a6c-9c30-61205bbe42ba" path="/var/lib/kubelet/pods/8a176e76-2a86-4a6c-9c30-61205bbe42ba/volumes"
Jul 27 16:31:37.707355 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707319    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:31:37.707813 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707797    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a869756b-781c-46f9-8c06-63d42f6c2765" containerName="maas-api"
Jul 27 16:31:37.707864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707815    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a869756b-781c-46f9-8c06-63d42f6c2765" containerName="maas-api"
Jul 27 16:31:37.707864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707849    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a176e76-2a86-4a6c-9c30-61205bbe42ba" containerName="revoke-keys"
Jul 27 16:31:37.707864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707858    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a176e76-2a86-4a6c-9c30-61205bbe42ba" containerName="revoke-keys"
Jul 27 16:31:37.707970 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707926    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a869756b-781c-46f9-8c06-63d42f6c2765" containerName="maas-api"
Jul 27 16:31:37.707970 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.707939    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a176e76-2a86-4a6c-9c30-61205bbe42ba" containerName="revoke-keys"
Jul 27 16:31:37.723126 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.723087    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:31:37.723400 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.723386    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.727603 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.727583    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-932843-serving-cert\""
Jul 27 16:31:37.734706 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.733443    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wf5r5\""
Jul 27 16:31:37.735258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.735240    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 27 16:31:37.735601 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.735585    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 16:31:37.871146 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.871087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.871317 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.871156    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.871317 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.871196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.971846 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.971758    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.971846 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.971821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.972029 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.971876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.972504 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.972480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.974454 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.974430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:37.982807 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:37.982782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v\") pod \"maas-api-e2e-shared-a-932843-c95667cb6-k6md2\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:38.038412 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.038373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:38.172208 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.172181    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:31:38.174443 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:31:38.174417    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a0c2ca_8181_4cf8_9a0b_841360413360.slice/crio-bb70b7426826ca26786b6fcc8ed62104277a0faaeea964c42671368a7df130d2 WatchSource:0}: Error finding container bb70b7426826ca26786b6fcc8ed62104277a0faaeea964c42671368a7df130d2: Status 404 returned error can't find the container with id bb70b7426826ca26786b6fcc8ed62104277a0faaeea964c42671368a7df130d2
Jul 27 16:31:38.885850 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.885805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" event={"ID":"85a0c2ca-8181-4cf8-9a0b-841360413360","Type":"ContainerStarted","Data":"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"}
Jul 27 16:31:38.886254 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.885876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" event={"ID":"85a0c2ca-8181-4cf8-9a0b-841360413360","Type":"ContainerStarted","Data":"bb70b7426826ca26786b6fcc8ed62104277a0faaeea964c42671368a7df130d2"}
Jul 27 16:31:38.886254 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.885909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:38.905190 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:38.905140    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" podStartSLOduration=1.305684955 podStartE2EDuration="1.905124253s" totalImagesPullingTime="599.439298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:31:38.903747339 +0000 UTC m=+2594.817803888" watchObservedRunningTime="2026-07-27 16:31:38.905124253 +0000 UTC m=+2594.819180815"
Jul 27 16:31:44.898648 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:44.898614    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:31:45.601358 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.601319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:31:45.750733 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.750689    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:31:45.754719 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.754697    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.757307 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.757277    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1f21d6-serving-cert\""
Jul 27 16:31:45.763998 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.763972    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:31:45.829280 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.829252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.829417 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.829294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.829417 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.829391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tts8\" (UniqueName: \"kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.929854 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.929783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.929854 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.929816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.930299 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.929930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tts8\" (UniqueName: \"kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.930299 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:31:45.930022    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-serving-cert: secret "maas-api-e2e-shared-b-1f21d6-serving-cert" not found
Jul 27 16:31:45.930299 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:31:45.930095    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls podName:652d348b-10f7-4326-978f-abacab8dc9a5 nodeName:}" failed. No retries permitted until 2026-07-27 16:31:46.430075555 +0000 UTC m=+2602.344132083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls") pod "maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" (UID: "652d348b-10f7-4326-978f-abacab8dc9a5") : secret "maas-api-e2e-shared-b-1f21d6-serving-cert" not found
Jul 27 16:31:45.930449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.930432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:45.939883 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:45.939858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tts8\" (UniqueName: \"kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:46.434229 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:46.434189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:46.437668 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:46.437639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") pod \"maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:46.667743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:46.667711    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:46.799508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:46.799482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:31:46.801368 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:31:46.801341    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652d348b_10f7_4326_978f_abacab8dc9a5.slice/crio-3c1f75d61c72fa4ebe2c5dd71762bca65f6317f2b36b0bcb97bd0ac144612556 WatchSource:0}: Error finding container 3c1f75d61c72fa4ebe2c5dd71762bca65f6317f2b36b0bcb97bd0ac144612556: Status 404 returned error can't find the container with id 3c1f75d61c72fa4ebe2c5dd71762bca65f6317f2b36b0bcb97bd0ac144612556
Jul 27 16:31:46.926114 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:46.926077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" event={"ID":"652d348b-10f7-4326-978f-abacab8dc9a5","Type":"ContainerStarted","Data":"3c1f75d61c72fa4ebe2c5dd71762bca65f6317f2b36b0bcb97bd0ac144612556"}
Jul 27 16:31:47.932121 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:47.932073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" event={"ID":"652d348b-10f7-4326-978f-abacab8dc9a5","Type":"ContainerStarted","Data":"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"}
Jul 27 16:31:47.932554 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:47.932164    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:47.954048 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:47.953998    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" podStartSLOduration=2.6279102290000003 podStartE2EDuration="2.953983824s" totalImagesPullingTime="326.073595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:31:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:31:47.951754139 +0000 UTC m=+2603.865810687" watchObservedRunningTime="2026-07-27 16:31:47.953983824 +0000 UTC m=+2603.868040372"
Jul 27 16:31:53.942587 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:53.942557    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:31:56.617015 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:31:56.616980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:32:18.203423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.203387    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:32:18.206990 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.206971    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.210854 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.210827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"kube-root-ca.crt\""
Jul 27 16:32:18.210972 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.210926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"default-dockercfg-xkl4d\""
Jul 27 16:32:18.212493 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.212476    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"openshift-service-ca.crt\""
Jul 27 16:32:18.212600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.212526    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"auth-test-model-932843-kserve-self-signed-certs\""
Jul 27 16:32:18.225670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.225646    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:32:18.322848 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.322816    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.322988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.322903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.322988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.322934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.322988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.322974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.323113 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.322994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.323113 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.323011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4dwq\" (UniqueName: \"kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.423797 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.423982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.423982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.423982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4dwq\" (UniqueName: \"kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.423982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.424225 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.423994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.424292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.424236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.424352 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.424302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.424408 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.424373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.426201 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.426184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.426472 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.426454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.433593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.433568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4dwq\" (UniqueName: \"kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq\") pod \"auth-test-model-932843-kserve-6994758d95-zrckf\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") " pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.515597 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.515531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:18.653902 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:18.653877    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:32:18.655029 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:32:18.655001    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50176081_d2dd_4767_8ed1_bf6e6acf71d3.slice/crio-a4610095f5468797eaba270ac9ab2186c8dd95c431ae87795dd542b35c6c04d4 WatchSource:0}: Error finding container a4610095f5468797eaba270ac9ab2186c8dd95c431ae87795dd542b35c6c04d4: Status 404 returned error can't find the container with id a4610095f5468797eaba270ac9ab2186c8dd95c431ae87795dd542b35c6c04d4
Jul 27 16:32:19.081819 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:19.081788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" event={"ID":"50176081-d2dd-4767-8ed1-bf6e6acf71d3","Type":"ContainerStarted","Data":"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"}
Jul 27 16:32:19.081819 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:19.081826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" event={"ID":"50176081-d2dd-4767-8ed1-bf6e6acf71d3","Type":"ContainerStarted","Data":"a4610095f5468797eaba270ac9ab2186c8dd95c431ae87795dd542b35c6c04d4"}
Jul 27 16:32:19.100738 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:19.100695    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" podStartSLOduration=1.100681116 podStartE2EDuration="1.100681116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:32:19.099311742 +0000 UTC m=+2635.013368289" watchObservedRunningTime="2026-07-27 16:32:19.100681116 +0000 UTC m=+2635.014737664"
Jul 27 16:32:21.603911 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:21.603879    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:32:28.516199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:28.516167    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:28.516199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:28.516205    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:28.529023 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:28.528997    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:29.137703 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:29.137674    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:32:43.702460 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.702427    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:32:43.705765 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.705747    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.709458 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.709437    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"openshift-service-ca.crt\""
Jul 27 16:32:43.709599 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.709472    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"kube-root-ca.crt\""
Jul 27 16:32:43.709599 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.709495    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"auth-test-model-1f21d6-kserve-self-signed-certs\""
Jul 27 16:32:43.709599 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.709442    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"default-dockercfg-28qjb\""
Jul 27 16:32:43.716698 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.716675    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:32:43.767953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.767759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.767953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.767835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.767953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.767890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5k82\" (UniqueName: \"kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.768288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.767996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.768288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.768021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.768288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.768057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869174 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869174 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869196    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869296    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5k82\" (UniqueName: \"kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869590 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869684 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.869741 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.869686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.871387 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.871370    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.871728 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.871713    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:43.878206 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:43.878185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5k82\" (UniqueName: \"kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82\") pod \"auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") " pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:44.015056 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:44.014968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:44.149566 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:44.149537    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:32:44.151635 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:32:44.151606    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67e48c71_e715_4fc0_bc3f_11cd19dec260.slice/crio-c1879cb167986f4dcddafd46215632bfb70a3d417099d5780b7b02ec4abbc7a0 WatchSource:0}: Error finding container c1879cb167986f4dcddafd46215632bfb70a3d417099d5780b7b02ec4abbc7a0: Status 404 returned error can't find the container with id c1879cb167986f4dcddafd46215632bfb70a3d417099d5780b7b02ec4abbc7a0
Jul 27 16:32:44.200457 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:44.200430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" event={"ID":"67e48c71-e715-4fc0-bc3f-11cd19dec260","Type":"ContainerStarted","Data":"c1879cb167986f4dcddafd46215632bfb70a3d417099d5780b7b02ec4abbc7a0"}
Jul 27 16:32:45.205593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:45.205555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" event={"ID":"67e48c71-e715-4fc0-bc3f-11cd19dec260","Type":"ContainerStarted","Data":"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"}
Jul 27 16:32:45.225333 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:45.225285    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" podStartSLOduration=2.225269416 podStartE2EDuration="2.225269416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:32:45.223960493 +0000 UTC m=+2661.138017041" watchObservedRunningTime="2026-07-27 16:32:45.225269416 +0000 UTC m=+2661.139325964"
Jul 27 16:32:47.000933 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:47.000898    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:32:54.015279 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:54.015247    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:54.015660 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:54.015289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:54.027613 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:54.027589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:32:54.270944 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:32:54.270867    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:33:17.007024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:33:17.006938    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:33:24.690191 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:33:24.690162    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:33:24.699073 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:33:24.699051    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:33:38.408952 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:33:38.408916    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:33:51.201214 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:33:51.201174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:34:11.831080 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:34:11.831046    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:34:29.793119 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:34:29.793079    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:34:57.413370 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:34:57.413290    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:35:14.796936 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:35:14.796899    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:35:37.202805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:35:37.202766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:35:55.402953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:35:55.402917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:36:18.197258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:36:18.197170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:36:25.097473 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:36:25.097443    2576 scope.go:117] "RemoveContainer" containerID="921ac8985860f612cbfb17961bf84b3548de718a25ef532860e51dc978c0e72d"
Jul 27 16:36:31.897641 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:36:31.897602    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:36:43.406437 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:36:43.406399    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:37:01.406398 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:01.406365    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:37:05.894505 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:05.894463    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:37:06.803403 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:06.803370    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:37:15.903643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.903611    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:37:15.906989 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.906972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.912593 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.912573    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"sub-test-model-932843-kserve-self-signed-certs\""
Jul 27 16:37:15.918162 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.918139    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:37:15.970981 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.970951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.971125 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.970986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.971125 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.971037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.971125 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.971077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.971263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.971160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2lb\" (UniqueName: \"kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:15.971263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:15.971196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072491 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2lb\" (UniqueName: \"kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.072805 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.072717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.073033 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.073007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.073187 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.073051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.073187 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.073138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.074930 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.074908    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.075209 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.075194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.081520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.081497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2lb\" (UniqueName: \"kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb\") pod \"sub-test-model-932843-kserve-778746c6b9-z2glh\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") " pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.215549 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.215520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:16.349603 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.349570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:37:16.350247 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:37:16.350222    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7aa7af22_4283_4494_a371_700f5d8b8bf2.slice/crio-bd1efacee62d1d03c738f3445ad8427ef4720bf646b024d5e5bffe09b6e3eaee WatchSource:0}: Error finding container bd1efacee62d1d03c738f3445ad8427ef4720bf646b024d5e5bffe09b6e3eaee: Status 404 returned error can't find the container with id bd1efacee62d1d03c738f3445ad8427ef4720bf646b024d5e5bffe09b6e3eaee
Jul 27 16:37:16.509413 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.509327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" event={"ID":"7aa7af22-4283-4494-a371-700f5d8b8bf2","Type":"ContainerStarted","Data":"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"}
Jul 27 16:37:16.509413 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.509370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" event={"ID":"7aa7af22-4283-4494-a371-700f5d8b8bf2","Type":"ContainerStarted","Data":"bd1efacee62d1d03c738f3445ad8427ef4720bf646b024d5e5bffe09b6e3eaee"}
Jul 27 16:37:16.530069 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:16.530016    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" podStartSLOduration=1.5300000599999999 podStartE2EDuration="1.53000006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:37:16.527639182 +0000 UTC m=+2932.441695731" watchObservedRunningTime="2026-07-27 16:37:16.53000006 +0000 UTC m=+2932.444056605"
Jul 27 16:37:19.692672 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:19.692637    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:37:20.913125 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:20.913068    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" podUID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" containerName="main" containerID="cri-o://711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2" gracePeriod=45
Jul 27 16:37:21.178543 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.178520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:37:21.223195 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223170    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223262    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223294    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223493 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223343    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223493 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223379    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4dwq\" (UniqueName: \"kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq\") pod \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\" (UID: \"50176081-d2dd-4767-8ed1-bf6e6acf71d3\") "
Jul 27 16:37:21.223592 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:21.223592 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home" (OuterVolumeSpecName: "home") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:21.223701 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223629    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:21.223769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223745    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.223821 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223771    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.223821 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.223785    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.225519 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.225483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm" (OuterVolumeSpecName: "dshm") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:21.225632 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.225570    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:37:21.225632 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.225617    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq" (OuterVolumeSpecName: "kube-api-access-p4dwq") pod "50176081-d2dd-4767-8ed1-bf6e6acf71d3" (UID: "50176081-d2dd-4767-8ed1-bf6e6acf71d3"). InnerVolumeSpecName "kube-api-access-p4dwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:37:21.325125 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.325087    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4dwq\" (UniqueName: \"kubernetes.io/projected/50176081-d2dd-4767-8ed1-bf6e6acf71d3-kube-api-access-p4dwq\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.325232 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.325126    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50176081-d2dd-4767-8ed1-bf6e6acf71d3-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.325232 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.325147    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50176081-d2dd-4767-8ed1-bf6e6acf71d3-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:21.537067 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.537033    2576 generic.go:356] "Generic (PLEG): container finished" podID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" containerID="711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2" exitCode=0
Jul 27 16:37:21.537263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.537089    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"
Jul 27 16:37:21.537263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.537095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" event={"ID":"50176081-d2dd-4767-8ed1-bf6e6acf71d3","Type":"ContainerDied","Data":"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"}
Jul 27 16:37:21.537263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.537154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf" event={"ID":"50176081-d2dd-4767-8ed1-bf6e6acf71d3","Type":"ContainerDied","Data":"a4610095f5468797eaba270ac9ab2186c8dd95c431ae87795dd542b35c6c04d4"}
Jul 27 16:37:21.537263 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.537174    2576 scope.go:117] "RemoveContainer" containerID="711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"
Jul 27 16:37:21.546782 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.546766    2576 scope.go:117] "RemoveContainer" containerID="711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"
Jul 27 16:37:21.547014 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:37:21.546996    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2\": container with ID starting with 711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2 not found: ID does not exist" containerID="711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"
Jul 27 16:37:21.547068 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.547021    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2"} err="failed to get container status \"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2\": rpc error: code = NotFound desc = could not find container \"711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2\": container with ID starting with 711ab0350d07021adb1cb8469151e566dbd10d2ae4fde63cea8f9ab80b4439c2 not found: ID does not exist"
Jul 27 16:37:21.560421 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.560394    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:37:21.565115 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.565088    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-932843/auth-test-model-932843-kserve-6994758d95-zrckf"]
Jul 27 16:37:21.822484 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:21.822413    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" podUID="67e48c71-e715-4fc0-bc3f-11cd19dec260" containerName="main" containerID="cri-o://da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340" gracePeriod=45
Jul 27 16:37:22.075617 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.075556    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:37:22.132668 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.132858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132677    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.132858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132713    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.132985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132893    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5k82\" (UniqueName: \"kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.132985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132956    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache" (OuterVolumeSpecName: "model-cache") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:22.133095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132976    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:22.133095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.132985    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.133095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.133070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm\") pod \"67e48c71-e715-4fc0-bc3f-11cd19dec260\" (UID: \"67e48c71-e715-4fc0-bc3f-11cd19dec260\") "
Jul 27 16:37:22.133272 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.133160    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home" (OuterVolumeSpecName: "home") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:22.133669 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.133642    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.133801 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.133671    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.133801 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.133688    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.135208 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.135183    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm" (OuterVolumeSpecName: "dshm") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:37:22.135455 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.135436    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82" (OuterVolumeSpecName: "kube-api-access-j5k82") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "kube-api-access-j5k82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:37:22.135520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.135502    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67e48c71-e715-4fc0-bc3f-11cd19dec260" (UID: "67e48c71-e715-4fc0-bc3f-11cd19dec260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:37:22.234995 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.234958    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67e48c71-e715-4fc0-bc3f-11cd19dec260-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.234995 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.234990    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67e48c71-e715-4fc0-bc3f-11cd19dec260-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.235193 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.235005    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5k82\" (UniqueName: \"kubernetes.io/projected/67e48c71-e715-4fc0-bc3f-11cd19dec260-kube-api-access-j5k82\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:37:22.544670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.544627    2576 generic.go:356] "Generic (PLEG): container finished" podID="67e48c71-e715-4fc0-bc3f-11cd19dec260" containerID="da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340" exitCode=0
Jul 27 16:37:22.544818 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.544680    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"
Jul 27 16:37:22.544818 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.544708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" event={"ID":"67e48c71-e715-4fc0-bc3f-11cd19dec260","Type":"ContainerDied","Data":"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"}
Jul 27 16:37:22.544818 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.544751    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p" event={"ID":"67e48c71-e715-4fc0-bc3f-11cd19dec260","Type":"ContainerDied","Data":"c1879cb167986f4dcddafd46215632bfb70a3d417099d5780b7b02ec4abbc7a0"}
Jul 27 16:37:22.544818 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.544770    2576 scope.go:117] "RemoveContainer" containerID="da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"
Jul 27 16:37:22.554138 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.554124    2576 scope.go:117] "RemoveContainer" containerID="da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"
Jul 27 16:37:22.554370 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:37:22.554349    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340\": container with ID starting with da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340 not found: ID does not exist" containerID="da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"
Jul 27 16:37:22.554433 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.554379    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340"} err="failed to get container status \"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340\": rpc error: code = NotFound desc = could not find container \"da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340\": container with ID starting with da1ba093562d6c2fb26be61da6d25f08a5627b12ca2772d11e06c8a884e50340 not found: ID does not exist"
Jul 27 16:37:22.567943 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.567917    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:37:22.572637 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.572617    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/auth-test-model-1f21d6-kserve-dd96cc4c9-gq67p"]
Jul 27 16:37:22.651797 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.651770    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" path="/var/lib/kubelet/pods/50176081-d2dd-4767-8ed1-bf6e6acf71d3/volumes"
Jul 27 16:37:22.652189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:22.652173    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e48c71-e715-4fc0-bc3f-11cd19dec260" path="/var/lib/kubelet/pods/67e48c71-e715-4fc0-bc3f-11cd19dec260/volumes"
Jul 27 16:37:25.115860 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:25.115827    2576 scope.go:117] "RemoveContainer" containerID="e98481965987494c9b482d13148dcbe43f710041e6aaf08c31e191ccd386be33"
Jul 27 16:37:25.124036 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:25.124015    2576 scope.go:117] "RemoveContainer" containerID="3bae6eb7b6f590536e7584bbce5e68b4688da5ca4b897de24b9acadef9f1024d"
Jul 27 16:37:26.216604 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:26.216555    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:26.216604 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:26.216613    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:26.229075 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:26.229047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:26.575190 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:26.575118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:37:30.792818 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.792443    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.792964    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" containerName="main"
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.792979    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" containerName="main"
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.792988    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e48c71-e715-4fc0-bc3f-11cd19dec260" containerName="main"
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.792994    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e48c71-e715-4fc0-bc3f-11cd19dec260" containerName="main"
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.793055    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="50176081-d2dd-4767-8ed1-bf6e6acf71d3" containerName="main"
Jul 27 16:37:30.793289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.793063    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e48c71-e715-4fc0-bc3f-11cd19dec260" containerName="main"
Jul 27 16:37:30.798182 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.798158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.802070 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.802026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"default-dockercfg-28qjb\""
Jul 27 16:37:30.802070 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.802058    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"openshift-service-ca.crt\""
Jul 27 16:37:30.802288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.802059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"sub-test-model-1f21d6-kserve-self-signed-certs\""
Jul 27 16:37:30.802288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.802135    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"kube-root-ca.crt\""
Jul 27 16:37:30.806383 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.806361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:37:30.911817 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.911781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.911985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.911827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.911985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.911901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.911985 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.911954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.912155 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.912028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:30.912155 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:30.912044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.012682 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.012856 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012738    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.012856 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.012856 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.013065 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.013065 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.012952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.013217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.013082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.013217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.013145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.013292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.013229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.015032 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.015011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.015383 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.015363    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.022073 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.022051    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28\") pod \"sub-test-model-1f21d6-kserve-5bf9d46659-npjf7\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") " pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.110173 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.110055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:31.248529 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.248503    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:37:31.250600 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:37:31.250574    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88248246_5e6d_4389_9408_38726047cca3.slice/crio-331f626d4e4cdc11cb074c476495c6573fc74f6de80bf5897d571c21215acf9c WatchSource:0}: Error finding container 331f626d4e4cdc11cb074c476495c6573fc74f6de80bf5897d571c21215acf9c: Status 404 returned error can't find the container with id 331f626d4e4cdc11cb074c476495c6573fc74f6de80bf5897d571c21215acf9c
Jul 27 16:37:31.587545 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.587514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" event={"ID":"88248246-5e6d-4389-9408-38726047cca3","Type":"ContainerStarted","Data":"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"}
Jul 27 16:37:31.587736 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.587555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" event={"ID":"88248246-5e6d-4389-9408-38726047cca3","Type":"ContainerStarted","Data":"331f626d4e4cdc11cb074c476495c6573fc74f6de80bf5897d571c21215acf9c"}
Jul 27 16:37:31.606988 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:31.606938    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" podStartSLOduration=1.606922663 podStartE2EDuration="1.606922663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:37:31.605298834 +0000 UTC m=+2947.519355394" watchObservedRunningTime="2026-07-27 16:37:31.606922663 +0000 UTC m=+2947.520979212"
Jul 27 16:37:35.594589 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:35.594559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:37:41.111224 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:41.111174    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:41.111224 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:41.111242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:41.124257 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:41.124229    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:37:41.645823 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:37:41.645781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:38:10.299547 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:10.299470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:38:24.715674 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:24.715640    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:38:24.725088 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:24.725070    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:38:28.702481 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:28.702447    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:38:35.568647 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:35.568614    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:38:36.479794 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:36.479761    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:38:42.591597 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.591563    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:38:42.595433 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.595412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.597918 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.597892    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-932843\"/\"rate-test-model-932843-kserve-self-signed-certs\""
Jul 27 16:38:42.604753 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.604732    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:38:42.660229 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.660369 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.660369 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.660369 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.660499 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2h2\" (UniqueName: \"kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.660499 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.660404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761062 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2h2\" (UniqueName: \"kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761176    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761262 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761568 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761610    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.761756 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.761657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.763517 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.763492    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.763680 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.763662    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.770184 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.770165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2h2\" (UniqueName: \"kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2\") pod \"rate-test-model-932843-kserve-cb6c886f9-px8zn\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") " pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:42.906468 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:42.906385    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:43.046553 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:43.046527    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:38:43.048259 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:38:43.048229    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31bedaf9_edd7_4684_afca_d8c2bf9b3222.slice/crio-627434bc9bb5632540639dfdc7f8774a08dec0265f6bae721946ac46a0f10140 WatchSource:0}: Error finding container 627434bc9bb5632540639dfdc7f8774a08dec0265f6bae721946ac46a0f10140: Status 404 returned error can't find the container with id 627434bc9bb5632540639dfdc7f8774a08dec0265f6bae721946ac46a0f10140
Jul 27 16:38:43.934999 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:43.934956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" event={"ID":"31bedaf9-edd7-4684-afca-d8c2bf9b3222","Type":"ContainerStarted","Data":"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"}
Jul 27 16:38:43.935415 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:43.935009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" event={"ID":"31bedaf9-edd7-4684-afca-d8c2bf9b3222","Type":"ContainerStarted","Data":"627434bc9bb5632540639dfdc7f8774a08dec0265f6bae721946ac46a0f10140"}
Jul 27 16:38:43.953953 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:43.953905    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" podStartSLOduration=1.953890167 podStartE2EDuration="1.953890167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:38:43.952569762 +0000 UTC m=+3019.866626323" watchObservedRunningTime="2026-07-27 16:38:43.953890167 +0000 UTC m=+3019.867946709"
Jul 27 16:38:47.601919 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:47.601876    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:38:50.588380 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.588319    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" podUID="7aa7af22-4283-4494-a371-700f5d8b8bf2" containerName="main" containerID="cri-o://01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46" gracePeriod=45
Jul 27 16:38:50.847650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.847592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:38:50.935883 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.935852    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936074 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.935946    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936074 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.935980    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936074 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936074 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936322 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng2lb\" (UniqueName: \"kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb\") pod \"7aa7af22-4283-4494-a371-700f5d8b8bf2\" (UID: \"7aa7af22-4283-4494-a371-700f5d8b8bf2\") "
Jul 27 16:38:50.936322 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936082    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home" (OuterVolumeSpecName: "home") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:50.936322 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936191    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:50.936474 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936412    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:50.936474 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936433    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:50.936650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.936609    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:50.938683 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.938650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm" (OuterVolumeSpecName: "dshm") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:50.938787 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.938772    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:38:50.938893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.938871    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb" (OuterVolumeSpecName: "kube-api-access-ng2lb") pod "7aa7af22-4283-4494-a371-700f5d8b8bf2" (UID: "7aa7af22-4283-4494-a371-700f5d8b8bf2"). InnerVolumeSpecName "kube-api-access-ng2lb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:38:50.971077 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.971044    2576 generic.go:356] "Generic (PLEG): container finished" podID="7aa7af22-4283-4494-a371-700f5d8b8bf2" containerID="01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46" exitCode=0
Jul 27 16:38:50.971242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.971092    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"
Jul 27 16:38:50.971242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.971148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" event={"ID":"7aa7af22-4283-4494-a371-700f5d8b8bf2","Type":"ContainerDied","Data":"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"}
Jul 27 16:38:50.971242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.971191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh" event={"ID":"7aa7af22-4283-4494-a371-700f5d8b8bf2","Type":"ContainerDied","Data":"bd1efacee62d1d03c738f3445ad8427ef4720bf646b024d5e5bffe09b6e3eaee"}
Jul 27 16:38:50.971242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.971212    2576 scope.go:117] "RemoveContainer" containerID="01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"
Jul 27 16:38:50.979893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.979877    2576 scope.go:117] "RemoveContainer" containerID="01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"
Jul 27 16:38:50.980184 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:38:50.980162    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46\": container with ID starting with 01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46 not found: ID does not exist" containerID="01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"
Jul 27 16:38:50.980242 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.980193    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46"} err="failed to get container status \"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46\": rpc error: code = NotFound desc = could not find container \"01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46\": container with ID starting with 01f70147dc1ee0d77333716117f733c08650c149432737e00822d194fb71aa46 not found: ID does not exist"
Jul 27 16:38:50.994021 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.993997    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:38:50.999646 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:50.999625    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-932843/sub-test-model-932843-kserve-778746c6b9-z2glh"]
Jul 27 16:38:51.037319 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.037295    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.037319 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.037319    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7aa7af22-4283-4494-a371-700f5d8b8bf2-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.037319 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.037327    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7aa7af22-4283-4494-a371-700f5d8b8bf2-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.037564 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.037336    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng2lb\" (UniqueName: \"kubernetes.io/projected/7aa7af22-4283-4494-a371-700f5d8b8bf2-kube-api-access-ng2lb\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.498565 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.498526    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" podUID="88248246-5e6d-4389-9408-38726047cca3" containerName="main" containerID="cri-o://4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1" gracePeriod=45
Jul 27 16:38:51.635342 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.635299    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" podUID="88248246-5e6d-4389-9408-38726047cca3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 27 16:38:51.751486 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.751422    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:38:51.844325 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844296    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844497 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844342    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844497 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844367    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844497 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844497 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844469    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844713 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844572    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home\") pod \"88248246-5e6d-4389-9408-38726047cca3\" (UID: \"88248246-5e6d-4389-9408-38726047cca3\") "
Jul 27 16:38:51.844713 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:51.844713 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache" (OuterVolumeSpecName: "model-cache") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:51.844846 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844800    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home" (OuterVolumeSpecName: "home") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:51.844893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844877    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.844945 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844893    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.844945 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.844902    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.846576 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.846554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:38:51.846698 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.846615    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28" (OuterVolumeSpecName: "kube-api-access-kdj28") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "kube-api-access-kdj28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:38:51.846698 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.846629    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm" (OuterVolumeSpecName: "dshm") pod "88248246-5e6d-4389-9408-38726047cca3" (UID: "88248246-5e6d-4389-9408-38726047cca3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:38:51.945983 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.945954    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdj28\" (UniqueName: \"kubernetes.io/projected/88248246-5e6d-4389-9408-38726047cca3-kube-api-access-kdj28\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.945983 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.945980    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88248246-5e6d-4389-9408-38726047cca3-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.945983 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.945990    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88248246-5e6d-4389-9408-38726047cca3-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:38:51.978736 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.978705    2576 generic.go:356] "Generic (PLEG): container finished" podID="88248246-5e6d-4389-9408-38726047cca3" containerID="4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1" exitCode=0
Jul 27 16:38:51.978863 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.978746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"
Jul 27 16:38:51.978863 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.978788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" event={"ID":"88248246-5e6d-4389-9408-38726047cca3","Type":"ContainerDied","Data":"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"}
Jul 27 16:38:51.978863 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.978829    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7" event={"ID":"88248246-5e6d-4389-9408-38726047cca3","Type":"ContainerDied","Data":"331f626d4e4cdc11cb074c476495c6573fc74f6de80bf5897d571c21215acf9c"}
Jul 27 16:38:51.978863 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.978847    2576 scope.go:117] "RemoveContainer" containerID="4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"
Jul 27 16:38:51.987498 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.987479    2576 scope.go:117] "RemoveContainer" containerID="4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"
Jul 27 16:38:51.987736 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:38:51.987717    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1\": container with ID starting with 4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1 not found: ID does not exist" containerID="4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"
Jul 27 16:38:51.987780 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.987744    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1"} err="failed to get container status \"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1\": rpc error: code = NotFound desc = could not find container \"4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1\": container with ID starting with 4e14c3401caca0bc4fb2a7029ff79c0ffb21e68e07775207803472e03f035de1 not found: ID does not exist"
Jul 27 16:38:51.999411 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:51.999385    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:38:52.003563 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.003510    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/sub-test-model-1f21d6-kserve-5bf9d46659-npjf7"]
Jul 27 16:38:52.653801 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.653765    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa7af22-4283-4494-a371-700f5d8b8bf2" path="/var/lib/kubelet/pods/7aa7af22-4283-4494-a371-700f5d8b8bf2/volumes"
Jul 27 16:38:52.654380 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.654359    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88248246-5e6d-4389-9408-38726047cca3" path="/var/lib/kubelet/pods/88248246-5e6d-4389-9408-38726047cca3/volumes"
Jul 27 16:38:52.906600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.906513    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:52.906600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.906551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:52.918792 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.918767    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:38:52.995237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:38:52.995213    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:39:23.393886 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.393845    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394213    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88248246-5e6d-4389-9408-38726047cca3" containerName="main"
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394224    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88248246-5e6d-4389-9408-38726047cca3" containerName="main"
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394240    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa7af22-4283-4494-a371-700f5d8b8bf2" containerName="main"
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394245    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa7af22-4283-4494-a371-700f5d8b8bf2" containerName="main"
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394300    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88248246-5e6d-4389-9408-38726047cca3" containerName="main"
Jul 27 16:39:23.394346 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.394307    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa7af22-4283-4494-a371-700f5d8b8bf2" containerName="main"
Jul 27 16:39:23.397692 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.397671    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.401252 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.401227    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"default-dockercfg-28qjb\""
Jul 27 16:39:23.401412 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.401294    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"rate-test-model-1f21d6-kserve-self-signed-certs\""
Jul 27 16:39:23.401508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.401495    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"kube-root-ca.crt\""
Jul 27 16:39:23.401508 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.401502    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1f21d6\"/\"openshift-service-ca.crt\""
Jul 27 16:39:23.408619 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.408597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:39:23.420900 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.420879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tpn\" (UniqueName: \"kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.421008 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.420928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.421008 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.420954    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.421008 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.420997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.421170 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.421026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.421170 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.421093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522479 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522449    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tpn\" (UniqueName: \"kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522657 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.522978 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.523044 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.522996    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.523093 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.523039    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.524834 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.524817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.525008 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.524990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.538117 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.538084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tpn\" (UniqueName: \"kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn\") pod \"rate-test-model-1f21d6-kserve-8558c5ccbc-djkch\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") " pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.708441 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.708409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:23.839394 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:23.839369    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:39:23.842231 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:39:23.842203    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod375fa890_4632_4ea9_9c36_8a1f5af8db34.slice/crio-5dc51497b5d0de33743051e4f7c18b5034f33a43ac9375185a25a493a084a13b WatchSource:0}: Error finding container 5dc51497b5d0de33743051e4f7c18b5034f33a43ac9375185a25a493a084a13b: Status 404 returned error can't find the container with id 5dc51497b5d0de33743051e4f7c18b5034f33a43ac9375185a25a493a084a13b
Jul 27 16:39:24.138600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:24.138512    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" event={"ID":"375fa890-4632-4ea9-9c36-8a1f5af8db34","Type":"ContainerStarted","Data":"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"}
Jul 27 16:39:24.138600 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:24.138555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" event={"ID":"375fa890-4632-4ea9-9c36-8a1f5af8db34","Type":"ContainerStarted","Data":"5dc51497b5d0de33743051e4f7c18b5034f33a43ac9375185a25a493a084a13b"}
Jul 27 16:39:24.157288 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:24.157239    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" podStartSLOduration=1.157223865 podStartE2EDuration="1.157223865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:39:24.154743849 +0000 UTC m=+3060.068800399" watchObservedRunningTime="2026-07-27 16:39:24.157223865 +0000 UTC m=+3060.071280412"
Jul 27 16:39:27.544421 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:27.544385    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:39:33.708780 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:33.708742    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:33.709239 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:33.708821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:33.721276 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:33.721254    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:39:34.196494 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:39:34.196470    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:40:00.202969 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:00.202935    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:40:15.205934 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:15.205891    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:40:30.001886 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:30.001851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:40:35.685423 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:35.685381    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:40:36.587944 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:36.587905    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:40:46.499371 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:46.499295    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:40:50.704468 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:50.704430    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" podUID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" containerName="main" containerID="cri-o://365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8" gracePeriod=45
Jul 27 16:40:50.949840 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:50.949817    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:40:51.016810 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016740    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz2h2\" (UniqueName: \"kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.016810 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016792    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.016973 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016843    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.016973 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016882    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.016973 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.016973 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.016943    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir\") pod \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\" (UID: \"31bedaf9-edd7-4684-afca-d8c2bf9b3222\") "
Jul 27 16:40:51.017199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.017059    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache" (OuterVolumeSpecName: "model-cache") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.017199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.017173    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home" (OuterVolumeSpecName: "home") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.017312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.017239    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.017312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.017258    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.017388 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.017296    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.018902 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.018881    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm" (OuterVolumeSpecName: "dshm") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.019024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.019003    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2" (OuterVolumeSpecName: "kube-api-access-gz2h2") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "kube-api-access-gz2h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:40:51.019074 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.019021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31bedaf9-edd7-4684-afca-d8c2bf9b3222" (UID: "31bedaf9-edd7-4684-afca-d8c2bf9b3222"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:40:51.118363 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.118334    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.118363 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.118360    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.118363 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.118370    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31bedaf9-edd7-4684-afca-d8c2bf9b3222-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.118562 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.118381    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz2h2\" (UniqueName: \"kubernetes.io/projected/31bedaf9-edd7-4684-afca-d8c2bf9b3222-kube-api-access-gz2h2\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.557918 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.557881    2576 generic.go:356] "Generic (PLEG): container finished" podID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" containerID="365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8" exitCode=0
Jul 27 16:40:51.558133 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.557927    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"
Jul 27 16:40:51.558133 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.557964    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" event={"ID":"31bedaf9-edd7-4684-afca-d8c2bf9b3222","Type":"ContainerDied","Data":"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"}
Jul 27 16:40:51.558133 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.558005    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn" event={"ID":"31bedaf9-edd7-4684-afca-d8c2bf9b3222","Type":"ContainerDied","Data":"627434bc9bb5632540639dfdc7f8774a08dec0265f6bae721946ac46a0f10140"}
Jul 27 16:40:51.558133 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.558029    2576 scope.go:117] "RemoveContainer" containerID="365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"
Jul 27 16:40:51.567582 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.567567    2576 scope.go:117] "RemoveContainer" containerID="365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"
Jul 27 16:40:51.567829 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:40:51.567811    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8\": container with ID starting with 365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8 not found: ID does not exist" containerID="365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"
Jul 27 16:40:51.567897 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.567842    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8"} err="failed to get container status \"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8\": rpc error: code = NotFound desc = could not find container \"365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8\": container with ID starting with 365427c1e2aab341eec00cf66a75db53a0d434b1c91930c35042ee64c1143ca8 not found: ID does not exist"
Jul 27 16:40:51.589579 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.589552    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:40:51.593558 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.593536    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-932843/rate-test-model-932843-kserve-cb6c886f9-px8zn"]
Jul 27 16:40:51.606327 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.606300    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" podUID="375fa890-4632-4ea9-9c36-8a1f5af8db34" containerName="main" containerID="cri-o://64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d" gracePeriod=45
Jul 27 16:40:51.861408 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.861384    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:40:51.924066 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924040    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924066 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924118    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924149    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924313 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924206    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4tpn\" (UniqueName: \"kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn\") pod \"375fa890-4632-4ea9-9c36-8a1f5af8db34\" (UID: \"375fa890-4632-4ea9-9c36-8a1f5af8db34\") "
Jul 27 16:40:51.924583 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.924583 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924449    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home" (OuterVolumeSpecName: "home") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.924583 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924516    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache" (OuterVolumeSpecName: "model-cache") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:51.924723 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924613    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.924723 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924630    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.924723 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.924642    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:51.926339 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.926319    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:40:51.926690 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.926658    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn" (OuterVolumeSpecName: "kube-api-access-j4tpn") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "kube-api-access-j4tpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:40:51.926690 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:51.926684    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm" (OuterVolumeSpecName: "dshm") pod "375fa890-4632-4ea9-9c36-8a1f5af8db34" (UID: "375fa890-4632-4ea9-9c36-8a1f5af8db34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:40:52.025792 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.025762    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tpn\" (UniqueName: \"kubernetes.io/projected/375fa890-4632-4ea9-9c36-8a1f5af8db34-kube-api-access-j4tpn\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:52.025792 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.025788    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/375fa890-4632-4ea9-9c36-8a1f5af8db34-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:52.025792 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.025797    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/375fa890-4632-4ea9-9c36-8a1f5af8db34-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:40:52.565122 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.565074    2576 generic.go:356] "Generic (PLEG): container finished" podID="375fa890-4632-4ea9-9c36-8a1f5af8db34" containerID="64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d" exitCode=0
Jul 27 16:40:52.565312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.565147    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"
Jul 27 16:40:52.565312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.565137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" event={"ID":"375fa890-4632-4ea9-9c36-8a1f5af8db34","Type":"ContainerDied","Data":"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"}
Jul 27 16:40:52.565312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.565275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch" event={"ID":"375fa890-4632-4ea9-9c36-8a1f5af8db34","Type":"ContainerDied","Data":"5dc51497b5d0de33743051e4f7c18b5034f33a43ac9375185a25a493a084a13b"}
Jul 27 16:40:52.565312 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.565302    2576 scope.go:117] "RemoveContainer" containerID="64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"
Jul 27 16:40:52.573965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.573946    2576 scope.go:117] "RemoveContainer" containerID="64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"
Jul 27 16:40:52.574259 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:40:52.574239    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d\": container with ID starting with 64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d not found: ID does not exist" containerID="64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"
Jul 27 16:40:52.574337 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.574266    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d"} err="failed to get container status \"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d\": rpc error: code = NotFound desc = could not find container \"64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d\": container with ID starting with 64ea5d874fa85338ccce45cd7fa31af6f667c22e764fefc9a2a2a60eeea8061d not found: ID does not exist"
Jul 27 16:40:52.590930 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.590892    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:40:52.597601 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.597552    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1f21d6/rate-test-model-1f21d6-kserve-8558c5ccbc-djkch"]
Jul 27 16:40:52.652706 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.652670    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" path="/var/lib/kubelet/pods/31bedaf9-edd7-4684-afca-d8c2bf9b3222/volumes"
Jul 27 16:40:52.653170 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:40:52.653152    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="375fa890-4632-4ea9-9c36-8a1f5af8db34" path="/var/lib/kubelet/pods/375fa890-4632-4ea9-9c36-8a1f5af8db34/volumes"
Jul 27 16:41:00.560149 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560084    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:41:00.560656 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560641    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="375fa890-4632-4ea9-9c36-8a1f5af8db34" containerName="main"
Jul 27 16:41:00.560711 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560659    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="375fa890-4632-4ea9-9c36-8a1f5af8db34" containerName="main"
Jul 27 16:41:00.560711 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560681    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" containerName="main"
Jul 27 16:41:00.560711 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560689    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" containerName="main"
Jul 27 16:41:00.560809 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560764    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="31bedaf9-edd7-4684-afca-d8c2bf9b3222" containerName="main"
Jul 27 16:41:00.560809 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.560779    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="375fa890-4632-4ea9-9c36-8a1f5af8db34" containerName="main"
Jul 27 16:41:00.565417 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.565394    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.568033 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.568010    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ffa17b-serving-cert\""
Jul 27 16:41:00.576648 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.576624    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:41:00.603562 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.603528    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.603696 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.603597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.603753 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.603712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.704186 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.704153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.704371 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.704218    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.704371 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.704268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.704489 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:00.704446    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-serving-cert: secret "maas-api-e2e-ipp-a-ffa17b-serving-cert" not found
Jul 27 16:41:00.704538 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:00.704531    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls podName:5c47ea8c-8bbe-4d36-95c6-ba57a44ce577 nodeName:}" failed. No retries permitted until 2026-07-27 16:41:01.20450073 +0000 UTC m=+3157.118557257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls") pod "maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" (UID: "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577") : secret "maas-api-e2e-ipp-a-ffa17b-serving-cert" not found
Jul 27 16:41:00.704840 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.704820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:00.715826 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:00.715800    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:01.209180 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:01.209140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:01.209341 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:01.209295    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-serving-cert: secret "maas-api-e2e-ipp-a-ffa17b-serving-cert" not found
Jul 27 16:41:01.209387 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:01.209355    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls podName:5c47ea8c-8bbe-4d36-95c6-ba57a44ce577 nodeName:}" failed. No retries permitted until 2026-07-27 16:41:02.209336501 +0000 UTC m=+3158.123393029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls") pod "maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" (UID: "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577") : secret "maas-api-e2e-ipp-a-ffa17b-serving-cert" not found
Jul 27 16:41:02.218749 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.218715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:02.221216 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.221192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:02.377908 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.377879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:02.514743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.514715    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:41:02.516898 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:41:02.516871    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c47ea8c_8bbe_4d36_95c6_ba57a44ce577.slice/crio-e2b4350ac63c3f7fe86afd74185025448adeb0dd4b569a12eb7aba005a14de6e WatchSource:0}: Error finding container e2b4350ac63c3f7fe86afd74185025448adeb0dd4b569a12eb7aba005a14de6e: Status 404 returned error can't find the container with id e2b4350ac63c3f7fe86afd74185025448adeb0dd4b569a12eb7aba005a14de6e
Jul 27 16:41:02.518283 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.518260    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 16:41:02.616024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:02.615989    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" event={"ID":"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577","Type":"ContainerStarted","Data":"e2b4350ac63c3f7fe86afd74185025448adeb0dd4b569a12eb7aba005a14de6e"}
Jul 27 16:41:04.626483 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:04.626400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" event={"ID":"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577","Type":"ContainerStarted","Data":"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"}
Jul 27 16:41:04.626842 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:04.626470    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:04.645155 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:04.645082    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" podStartSLOduration=2.820085961 podStartE2EDuration="4.645064577s" totalImagesPullingTime="1.824978616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:41:04.644138288 +0000 UTC m=+3160.558194838" watchObservedRunningTime="2026-07-27 16:41:04.645064577 +0000 UTC m=+3160.559121126"
Jul 27 16:41:08.200201 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:08.200163    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:41:10.636453 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:10.636422    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:41:24.615812 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.615775    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:41:24.624095 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.624071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.626603 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.626574    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d32cda-serving-cert\""
Jul 27 16:41:24.629979 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.629914    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:41:24.716379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.716354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.716534 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.716389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.716534 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.716508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hh92\" (UniqueName: \"kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.817609 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.817578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hh92\" (UniqueName: \"kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.817755 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.817646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.817755 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.817680    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.817839 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:24.817785    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-serving-cert: secret "maas-api-e2e-ipp-b-d32cda-serving-cert" not found
Jul 27 16:41:24.817874 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:41:24.817864    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls podName:4d301137-a9f8-45c2-8919-ab90391fd737 nodeName:}" failed. No retries permitted until 2026-07-27 16:41:25.317844337 +0000 UTC m=+3181.231900867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls") pod "maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" (UID: "4d301137-a9f8-45c2-8919-ab90391fd737") : secret "maas-api-e2e-ipp-b-d32cda-serving-cert" not found
Jul 27 16:41:24.818425 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.818407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:24.828247 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:24.828219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hh92\" (UniqueName: \"kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:25.321743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:25.321703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:25.324773 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:25.324744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:25.537734 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:25.537700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:25.682790 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:25.682764    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:41:25.684556 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:41:25.684525    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d301137_a9f8_45c2_8919_ab90391fd737.slice/crio-4f0dfd080186369931e9a96b798ded78a1b421578301044b7c160e3e6d550db4 WatchSource:0}: Error finding container 4f0dfd080186369931e9a96b798ded78a1b421578301044b7c160e3e6d550db4: Status 404 returned error can't find the container with id 4f0dfd080186369931e9a96b798ded78a1b421578301044b7c160e3e6d550db4
Jul 27 16:41:25.735535 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:25.735500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" event={"ID":"4d301137-a9f8-45c2-8919-ab90391fd737","Type":"ContainerStarted","Data":"4f0dfd080186369931e9a96b798ded78a1b421578301044b7c160e3e6d550db4"}
Jul 27 16:41:26.741596 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:26.741554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" event={"ID":"4d301137-a9f8-45c2-8919-ab90391fd737","Type":"ContainerStarted","Data":"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"}
Jul 27 16:41:26.741979 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:26.741618    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:26.761112 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:26.761048    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" podStartSLOduration=2.354127433 podStartE2EDuration="2.761031369s" totalImagesPullingTime="406.903936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:41:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:41:26.759728569 +0000 UTC m=+3182.673785143" watchObservedRunningTime="2026-07-27 16:41:26.761031369 +0000 UTC m=+3182.675087917"
Jul 27 16:41:30.807092 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:30.807054    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:41:32.752024 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:32.751995    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:41:57.499030 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.498994    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:41:57.502766 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.502744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.505279 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.505255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ffa17b\"/\"kube-root-ca.crt\""
Jul 27 16:41:57.506493 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.506475    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ffa17b\"/\"default-dockercfg-j7gsx\""
Jul 27 16:41:57.506586 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.506535    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-ffa17b\"/\"ipp-route-ffa17b-kserve-self-signed-certs\""
Jul 27 16:41:57.506586 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.506481    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-ffa17b\"/\"openshift-service-ca.crt\""
Jul 27 16:41:57.510577 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.510555    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:41:57.625134 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.625284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.625284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.625284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.625284 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmfsf\" (UniqueName: \"kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.625422 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.625286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726296 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726312    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmfsf\" (UniqueName: \"kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726706 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726779 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.726779 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.727006 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.726977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.727006 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.727001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.728711 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.728689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.728983 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.728967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.734839 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.734817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmfsf\" (UniqueName: \"kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf\") pod \"ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") " pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.812822 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.812733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:41:57.945855 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:57.945834    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:41:57.948943 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:41:57.948915    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1b50b96_3e7f_464d_8927_bf992efc1583.slice/crio-fa12fd93889de91c795deba13ec5901370128fc4e4f7a31a4442bea11f026c21 WatchSource:0}: Error finding container fa12fd93889de91c795deba13ec5901370128fc4e4f7a31a4442bea11f026c21: Status 404 returned error can't find the container with id fa12fd93889de91c795deba13ec5901370128fc4e4f7a31a4442bea11f026c21
Jul 27 16:41:58.893089 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:58.893056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" event={"ID":"d1b50b96-3e7f-464d-8927-bf992efc1583","Type":"ContainerStarted","Data":"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"}
Jul 27 16:41:58.893089 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:58.893094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" event={"ID":"d1b50b96-3e7f-464d-8927-bf992efc1583","Type":"ContainerStarted","Data":"fa12fd93889de91c795deba13ec5901370128fc4e4f7a31a4442bea11f026c21"}
Jul 27 16:41:58.912009 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:41:58.911951    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" podStartSLOduration=1.911935566 podStartE2EDuration="1.911935566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:41:58.910823405 +0000 UTC m=+3214.824879950" watchObservedRunningTime="2026-07-27 16:41:58.911935566 +0000 UTC m=+3214.825992114"
Jul 27 16:42:00.992167 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:00.992128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:42:07.813841 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:07.813798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:42:07.814419 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:07.813856    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:42:07.826358 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:07.826332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:42:07.950792 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:07.950757    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:42:26.903157 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:26.903120    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:42:40.496924 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.496887    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:42:40.500944 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.500925    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.503512 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.503489    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-145981-serving-cert\""
Jul 27 16:42:40.510611 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.510585    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:42:40.623916 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.623888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.624070 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.623949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7h77\" (UniqueName: \"kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.624070 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.624024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.725112 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.725075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.725292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.725163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.725292 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.725224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7h77\" (UniqueName: \"kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.725416 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:42:40.725303    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-serving-cert: secret "maas-api-e2e-ipp-cleanup-145981-serving-cert" not found
Jul 27 16:42:40.725416 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:42:40.725365    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls podName:6e5f1ac8-737f-44bd-884b-07fbba488d95 nodeName:}" failed. No retries permitted until 2026-07-27 16:42:41.225347484 +0000 UTC m=+3257.139404010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" (UID: "6e5f1ac8-737f-44bd-884b-07fbba488d95") : secret "maas-api-e2e-ipp-cleanup-145981-serving-cert" not found
Jul 27 16:42:40.725675 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.725656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:40.735310 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:40.735284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7h77\" (UniqueName: \"kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:41.229903 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:41.229853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:41.232441 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:41.232417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:41.411813 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:41.411779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:41.547562 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:41.547538    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:42:41.550600 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:42:41.550571    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5f1ac8_737f_44bd_884b_07fbba488d95.slice/crio-e5e5287a7073f3614aaf12c761b4fe01e0ac913ccf1551e293cb01a0ad5b3cd3 WatchSource:0}: Error finding container e5e5287a7073f3614aaf12c761b4fe01e0ac913ccf1551e293cb01a0ad5b3cd3: Status 404 returned error can't find the container with id e5e5287a7073f3614aaf12c761b4fe01e0ac913ccf1551e293cb01a0ad5b3cd3
Jul 27 16:42:42.094419 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:42.094384    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" event={"ID":"6e5f1ac8-737f-44bd-884b-07fbba488d95","Type":"ContainerStarted","Data":"e5e5287a7073f3614aaf12c761b4fe01e0ac913ccf1551e293cb01a0ad5b3cd3"}
Jul 27 16:42:43.100719 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:43.100674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" event={"ID":"6e5f1ac8-737f-44bd-884b-07fbba488d95","Type":"ContainerStarted","Data":"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"}
Jul 27 16:42:43.101098 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:43.100742    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:43.119046 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:43.119000    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" podStartSLOduration=2.55984344 podStartE2EDuration="3.118970846s" totalImagesPullingTime="559.127406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:42:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:42:43.117311276 +0000 UTC m=+3259.031367825" watchObservedRunningTime="2026-07-27 16:42:43.118970846 +0000 UTC m=+3259.033027394"
Jul 27 16:42:45.195628 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:45.195591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:42:49.111302 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:49.111275    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:42:55.771676 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:55.771637    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"]
Jul 27 16:42:55.776433 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:55.776412    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:55.778804 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:55.778783    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:42:55.782518 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:55.782495    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"]
Jul 27 16:42:55.963538 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:55.963503    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq\" (UID: \"5bbeea25-f20a-4653-b0d4-d19d495a8fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:56.064790 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:56.064714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq\" (UID: \"5bbeea25-f20a-4653-b0d4-d19d495a8fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:56.065444 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:56.065422    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq\" (UID: \"5bbeea25-f20a-4653-b0d4-d19d495a8fe2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:56.088800 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:56.088775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:56.436541 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:56.436517    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"]
Jul 27 16:42:56.439056 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:42:56.439018    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbeea25_f20a_4653_b0d4_d19d495a8fe2.slice/crio-22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed WatchSource:0}: Error finding container 22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed: Status 404 returned error can't find the container with id 22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed
Jul 27 16:42:57.172946 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:57.172864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq" event={"ID":"5bbeea25-f20a-4653-b0d4-d19d495a8fe2","Type":"ContainerStarted","Data":"50f82a4dfd89fc6c00aa4443f77499cd1fc0e894dd17a9780a4a62e1dc71e6d8"}
Jul 27 16:42:57.172946 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:57.172920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq" event={"ID":"5bbeea25-f20a-4653-b0d4-d19d495a8fe2","Type":"ContainerStarted","Data":"22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed"}
Jul 27 16:42:57.188138 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:57.188071    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq" podStartSLOduration=2.188056803 podStartE2EDuration="2.188056803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:42:57.187397596 +0000 UTC m=+3273.101454144" watchObservedRunningTime="2026-07-27 16:42:57.188056803 +0000 UTC m=+3273.102113350"
Jul 27 16:42:58.179115 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:58.179066    2576 generic.go:356] "Generic (PLEG): container finished" podID="5bbeea25-f20a-4653-b0d4-d19d495a8fe2" containerID="50f82a4dfd89fc6c00aa4443f77499cd1fc0e894dd17a9780a4a62e1dc71e6d8" exitCode=0
Jul 27 16:42:58.179477 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:58.179157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq" event={"ID":"5bbeea25-f20a-4653-b0d4-d19d495a8fe2","Type":"ContainerDied","Data":"50f82a4dfd89fc6c00aa4443f77499cd1fc0e894dd17a9780a4a62e1dc71e6d8"}
Jul 27 16:42:59.322424 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:59.322397    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:42:59.495803 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:59.495775    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca\") pod \"5bbeea25-f20a-4653-b0d4-d19d495a8fe2\" (UID: \"5bbeea25-f20a-4653-b0d4-d19d495a8fe2\") "
Jul 27 16:42:59.496127 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:59.496089    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bbeea25-f20a-4653-b0d4-d19d495a8fe2" (UID: "5bbeea25-f20a-4653-b0d4-d19d495a8fe2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:42:59.597225 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:42:59.597197    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bbeea25-f20a-4653-b0d4-d19d495a8fe2-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:00.189992 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:00.189965    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"
Jul 27 16:43:00.189992 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:00.189988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq" event={"ID":"5bbeea25-f20a-4653-b0d4-d19d495a8fe2","Type":"ContainerDied","Data":"22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed"}
Jul 27 16:43:00.190235 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:00.190030    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22103f69add7328f9350f75c8d2535044c541cde95a7131ad864989b4df856ed"
Jul 27 16:43:06.186119 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.186067    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:43:06.186580 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.186331    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" podUID="6e5f1ac8-737f-44bd-884b-07fbba488d95" containerName="maas-api" containerID="cri-o://3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776" gracePeriod=30
Jul 27 16:43:06.448402 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.448373    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:43:06.458204 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.458182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7h77\" (UniqueName: \"kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77\") pod \"6e5f1ac8-737f-44bd-884b-07fbba488d95\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") "
Jul 27 16:43:06.458345 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.458236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") pod \"6e5f1ac8-737f-44bd-884b-07fbba488d95\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") "
Jul 27 16:43:06.458345 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.458253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca\") pod \"6e5f1ac8-737f-44bd-884b-07fbba488d95\" (UID: \"6e5f1ac8-737f-44bd-884b-07fbba488d95\") "
Jul 27 16:43:06.458628 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.458608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e5f1ac8-737f-44bd-884b-07fbba488d95" (UID: "6e5f1ac8-737f-44bd-884b-07fbba488d95"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:43:06.460245 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.460224    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77" (OuterVolumeSpecName: "kube-api-access-w7h77") pod "6e5f1ac8-737f-44bd-884b-07fbba488d95" (UID: "6e5f1ac8-737f-44bd-884b-07fbba488d95"). InnerVolumeSpecName "kube-api-access-w7h77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:43:06.460317 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.460272    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e5f1ac8-737f-44bd-884b-07fbba488d95" (UID: "6e5f1ac8-737f-44bd-884b-07fbba488d95"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:43:06.559816 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.559789    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7h77\" (UniqueName: \"kubernetes.io/projected/6e5f1ac8-737f-44bd-884b-07fbba488d95-kube-api-access-w7h77\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:06.559816 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.559814    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5f1ac8-737f-44bd-884b-07fbba488d95-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:06.559972 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:06.559823    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5f1ac8-737f-44bd-884b-07fbba488d95-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:07.222743 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.222710    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e5f1ac8-737f-44bd-884b-07fbba488d95" containerID="3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776" exitCode=0
Jul 27 16:43:07.223199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.222757    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"
Jul 27 16:43:07.223199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.222792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" event={"ID":"6e5f1ac8-737f-44bd-884b-07fbba488d95","Type":"ContainerDied","Data":"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"}
Jul 27 16:43:07.223199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.222862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s" event={"ID":"6e5f1ac8-737f-44bd-884b-07fbba488d95","Type":"ContainerDied","Data":"e5e5287a7073f3614aaf12c761b4fe01e0ac913ccf1551e293cb01a0ad5b3cd3"}
Jul 27 16:43:07.223199 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.222890    2576 scope.go:117] "RemoveContainer" containerID="3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"
Jul 27 16:43:07.231504 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.231485    2576 scope.go:117] "RemoveContainer" containerID="3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"
Jul 27 16:43:07.231725 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:43:07.231710    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776\": container with ID starting with 3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776 not found: ID does not exist" containerID="3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"
Jul 27 16:43:07.231782 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.231732    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776"} err="failed to get container status \"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776\": rpc error: code = NotFound desc = could not find container \"3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776\": container with ID starting with 3133113a457df39363093b39eb6f073fd52fe5738bbcd101f51942a8ebdb4776 not found: ID does not exist"
Jul 27 16:43:07.240281 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.240255    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:43:07.244572 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:07.244549    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-145981-84cf7cf49b-2x97s"]
Jul 27 16:43:08.652613 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:08.652578    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5f1ac8-737f-44bd-884b-07fbba488d95" path="/var/lib/kubelet/pods/6e5f1ac8-737f-44bd-884b-07fbba488d95/volumes"
Jul 27 16:43:10.702465 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:10.702430    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:43:10.847435 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:10.847403    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"]
Jul 27 16:43:10.849794 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:10.849770    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-145981-db8vq"]
Jul 27 16:43:12.652493 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:12.652463    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbeea25-f20a-4653-b0d4-d19d495a8fe2" path="/var/lib/kubelet/pods/5bbeea25-f20a-4653-b0d4-d19d495a8fe2/volumes"
Jul 27 16:43:21.369450 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369414    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"]
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369779    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5f1ac8-737f-44bd-884b-07fbba488d95" containerName="maas-api"
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369791    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5f1ac8-737f-44bd-884b-07fbba488d95" containerName="maas-api"
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369816    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbeea25-f20a-4653-b0d4-d19d495a8fe2" containerName="revoke-keys"
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369821    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbeea25-f20a-4653-b0d4-d19d495a8fe2" containerName="revoke-keys"
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369879    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbeea25-f20a-4653-b0d4-d19d495a8fe2" containerName="revoke-keys"
Jul 27 16:43:21.369928 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.369890    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5f1ac8-737f-44bd-884b-07fbba488d95" containerName="maas-api"
Jul 27 16:43:21.374135 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.374096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:21.376622 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.376606    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:43:21.382558 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.382532    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"]
Jul 27 16:43:21.492333 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.492298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5\" (UID: \"59eacba8-bd17-46f3-abeb-d1d8a85cf3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:21.593727 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.593687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5\" (UID: \"59eacba8-bd17-46f3-abeb-d1d8a85cf3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:21.594322 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.594302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5\" (UID: \"59eacba8-bd17-46f3-abeb-d1d8a85cf3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:21.685319 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.685235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:21.821828 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:21.821801    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"]
Jul 27 16:43:21.824363 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:43:21.824319    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59eacba8_bd17_46f3_abeb_d1d8a85cf3af.slice/crio-3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131 WatchSource:0}: Error finding container 3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131: Status 404 returned error can't find the container with id 3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131
Jul 27 16:43:22.304825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:22.304731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5" event={"ID":"59eacba8-bd17-46f3-abeb-d1d8a85cf3af","Type":"ContainerStarted","Data":"c6b791ef7ef21da0f4bc42d250fbba983bc2ededd95c44c85444b98097389fbe"}
Jul 27 16:43:22.304825 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:22.304784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5" event={"ID":"59eacba8-bd17-46f3-abeb-d1d8a85cf3af","Type":"ContainerStarted","Data":"3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131"}
Jul 27 16:43:22.324193 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:22.324146    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5" podStartSLOduration=1.32413086 podStartE2EDuration="1.32413086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:43:22.322732359 +0000 UTC m=+3298.236788905" watchObservedRunningTime="2026-07-27 16:43:22.32413086 +0000 UTC m=+3298.238187408"
Jul 27 16:43:23.311360 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:23.311322    2576 generic.go:356] "Generic (PLEG): container finished" podID="59eacba8-bd17-46f3-abeb-d1d8a85cf3af" containerID="c6b791ef7ef21da0f4bc42d250fbba983bc2ededd95c44c85444b98097389fbe" exitCode=0
Jul 27 16:43:23.311799 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:23.311404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5" event={"ID":"59eacba8-bd17-46f3-abeb-d1d8a85cf3af","Type":"ContainerDied","Data":"c6b791ef7ef21da0f4bc42d250fbba983bc2ededd95c44c85444b98097389fbe"}
Jul 27 16:43:24.449893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.449869    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:24.521617 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.521592    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca\") pod \"59eacba8-bd17-46f3-abeb-d1d8a85cf3af\" (UID: \"59eacba8-bd17-46f3-abeb-d1d8a85cf3af\") "
Jul 27 16:43:24.521886 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.521865    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59eacba8-bd17-46f3-abeb-d1d8a85cf3af" (UID: "59eacba8-bd17-46f3-abeb-d1d8a85cf3af"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:43:24.622952 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.622885    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59eacba8-bd17-46f3-abeb-d1d8a85cf3af-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:24.745536 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.745508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:43:24.755447 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:24.755427    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:43:25.321153 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:25.321120    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5" event={"ID":"59eacba8-bd17-46f3-abeb-d1d8a85cf3af","Type":"ContainerDied","Data":"3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131"}
Jul 27 16:43:25.321153 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:25.321142    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"
Jul 27 16:43:25.321381 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:25.321168    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3906950694d55faa92b5f36756a70ee093ed2cccd9957aba4b06b6b78a833131"
Jul 27 16:43:35.400529 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:35.400493    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:43:36.812139 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:36.812094    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:43:36.812528 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:36.812299    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" podUID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" containerName="maas-api" containerID="cri-o://1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0" gracePeriod=30
Jul 27 16:43:37.081080 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.081056    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:43:37.234097 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.234061    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") pod \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") "
Jul 27 16:43:37.234307 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.234138    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca\") pod \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") "
Jul 27 16:43:37.234307 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.234180    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v\") pod \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\" (UID: \"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577\") "
Jul 27 16:43:37.234873 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.234844    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" (UID: "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:43:37.236929 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.236900    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" (UID: "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:43:37.237227 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.237085    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v" (OuterVolumeSpecName: "kube-api-access-rnh4v") pod "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" (UID: "5c47ea8c-8bbe-4d36-95c6-ba57a44ce577"). InnerVolumeSpecName "kube-api-access-rnh4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:43:37.335469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.335372    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:37.335469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.335411    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:37.335469 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.335427    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnh4v\" (UniqueName: \"kubernetes.io/projected/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577-kube-api-access-rnh4v\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:43:37.378249 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.378212    2576 generic.go:356] "Generic (PLEG): container finished" podID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" containerID="1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0" exitCode=0
Jul 27 16:43:37.378448 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.378254    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"
Jul 27 16:43:37.378448 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.378293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" event={"ID":"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577","Type":"ContainerDied","Data":"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"}
Jul 27 16:43:37.378448 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.378348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv" event={"ID":"5c47ea8c-8bbe-4d36-95c6-ba57a44ce577","Type":"ContainerDied","Data":"e2b4350ac63c3f7fe86afd74185025448adeb0dd4b569a12eb7aba005a14de6e"}
Jul 27 16:43:37.378448 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.378377    2576 scope.go:117] "RemoveContainer" containerID="1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"
Jul 27 16:43:37.389216 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.389166    2576 scope.go:117] "RemoveContainer" containerID="1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"
Jul 27 16:43:37.389637 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:43:37.389614    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0\": container with ID starting with 1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0 not found: ID does not exist" containerID="1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"
Jul 27 16:43:37.389715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.389647    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0"} err="failed to get container status \"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0\": rpc error: code = NotFound desc = could not find container \"1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0\": container with ID starting with 1c25913dd56c5a675b58a3e3ff8efce1b50d4bb35158a3d4f6c57a1453002bf0 not found: ID does not exist"
Jul 27 16:43:37.400708 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.400687    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:43:37.405375 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:37.405356    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ffa17b-78dd47fc8b-7l6hv"]
Jul 27 16:43:38.653834 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:38.653799    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" path="/var/lib/kubelet/pods/5c47ea8c-8bbe-4d36-95c6-ba57a44ce577/volumes"
Jul 27 16:43:41.431680 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:41.431649    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"]
Jul 27 16:43:41.436212 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:41.436189    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ffa17b-brzf5"]
Jul 27 16:43:42.652880 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:42.652847    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59eacba8-bd17-46f3-abeb-d1d8a85cf3af" path="/var/lib/kubelet/pods/59eacba8-bd17-46f3-abeb-d1d8a85cf3af/volumes"
Jul 27 16:43:44.998337 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:44.998304    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:43:46.809913 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:43:46.809834    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:44:01.828845 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:01.828779    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" podUID="d1b50b96-3e7f-464d-8927-bf992efc1583" containerName="main" containerID="cri-o://115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c" gracePeriod=45
Jul 27 16:44:02.095190 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.095163    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:44:02.144946 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.144912    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145166 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145021    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145166 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145076    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmfsf\" (UniqueName: \"kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145166 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145362 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145166    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145362 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145179    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home" (OuterVolumeSpecName: "home") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:44:02.145362 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache\") pod \"d1b50b96-3e7f-464d-8927-bf992efc1583\" (UID: \"d1b50b96-3e7f-464d-8927-bf992efc1583\") "
Jul 27 16:44:02.145362 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145323    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:44:02.145591 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145481    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-tmp-dir\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.145591 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145498    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-home\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.145591 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.145511    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:44:02.147524 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.147499    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf" (OuterVolumeSpecName: "kube-api-access-mmfsf") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "kube-api-access-mmfsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:44:02.147652 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.147602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:44:02.147652 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.147616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm" (OuterVolumeSpecName: "dshm") pod "d1b50b96-3e7f-464d-8927-bf992efc1583" (UID: "d1b50b96-3e7f-464d-8927-bf992efc1583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 16:44:02.246280 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.246252    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmfsf\" (UniqueName: \"kubernetes.io/projected/d1b50b96-3e7f-464d-8927-bf992efc1583-kube-api-access-mmfsf\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.246280 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.246277    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1b50b96-3e7f-464d-8927-bf992efc1583-tls-certs\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.246280 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.246286    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-dshm\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.246482 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.246294    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1b50b96-3e7f-464d-8927-bf992efc1583-model-cache\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:02.498149 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.498095    2576 generic.go:356] "Generic (PLEG): container finished" podID="d1b50b96-3e7f-464d-8927-bf992efc1583" containerID="115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c" exitCode=0
Jul 27 16:44:02.498315 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.498155    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"
Jul 27 16:44:02.498315 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.498162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" event={"ID":"d1b50b96-3e7f-464d-8927-bf992efc1583","Type":"ContainerDied","Data":"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"}
Jul 27 16:44:02.498315 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.498201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64" event={"ID":"d1b50b96-3e7f-464d-8927-bf992efc1583","Type":"ContainerDied","Data":"fa12fd93889de91c795deba13ec5901370128fc4e4f7a31a4442bea11f026c21"}
Jul 27 16:44:02.498315 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.498225    2576 scope.go:117] "RemoveContainer" containerID="115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"
Jul 27 16:44:02.507520 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.507504    2576 scope.go:117] "RemoveContainer" containerID="115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"
Jul 27 16:44:02.507751 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:44:02.507731    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c\": container with ID starting with 115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c not found: ID does not exist" containerID="115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"
Jul 27 16:44:02.507817 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.507758    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c"} err="failed to get container status \"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c\": rpc error: code = NotFound desc = could not find container \"115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c\": container with ID starting with 115083c64d2e53f1ded994d7a771c9549a148672407b205a5529bcaada36c58c not found: ID does not exist"
Jul 27 16:44:02.521999 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.521976    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:44:02.526311 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.526291    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-ffa17b/ipp-route-ffa17b-kserve-5bc8bccbf9-lbs64"]
Jul 27 16:44:02.652965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:02.652938    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1b50b96-3e7f-464d-8927-bf992efc1583" path="/var/lib/kubelet/pods/d1b50b96-3e7f-464d-8927-bf992efc1583/volumes"
Jul 27 16:44:31.468405 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468371    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"]
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59eacba8-bd17-46f3-abeb-d1d8a85cf3af" containerName="revoke-keys"
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59eacba8-bd17-46f3-abeb-d1d8a85cf3af" containerName="revoke-keys"
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468778    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" containerName="maas-api"
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468783    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" containerName="maas-api"
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468790    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1b50b96-3e7f-464d-8927-bf992efc1583" containerName="main"
Jul 27 16:44:31.468808 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468795    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1b50b96-3e7f-464d-8927-bf992efc1583" containerName="main"
Jul 27 16:44:31.469039 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468845    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c47ea8c-8bbe-4d36-95c6-ba57a44ce577" containerName="maas-api"
Jul 27 16:44:31.469039 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468857    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59eacba8-bd17-46f3-abeb-d1d8a85cf3af" containerName="revoke-keys"
Jul 27 16:44:31.469039 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.468865    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1b50b96-3e7f-464d-8927-bf992efc1583" containerName="main"
Jul 27 16:44:31.473170 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.473153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:31.475842 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.475820    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:44:31.478931 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.478908    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"]
Jul 27 16:44:31.596549 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.596514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk\" (UID: \"21a2fb75-449d-481c-8968-2c81df36a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:31.697166 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.697130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk\" (UID: \"21a2fb75-449d-481c-8968-2c81df36a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:31.697752 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.697734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk\" (UID: \"21a2fb75-449d-481c-8968-2c81df36a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:31.785290 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.785211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:31.923962 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:31.923934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"]
Jul 27 16:44:31.926790 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:44:31.926751    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a2fb75_449d_481c_8968_2c81df36a40d.slice/crio-4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee WatchSource:0}: Error finding container 4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee: Status 404 returned error can't find the container with id 4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee
Jul 27 16:44:32.653566 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:32.653530    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk" event={"ID":"21a2fb75-449d-481c-8968-2c81df36a40d","Type":"ContainerStarted","Data":"1e8fb533d61fe40667a088fcba244d9d58713e3fb900ee9b30aabf779d281aaa"}
Jul 27 16:44:32.653932 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:32.653577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk" event={"ID":"21a2fb75-449d-481c-8968-2c81df36a40d","Type":"ContainerStarted","Data":"4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee"}
Jul 27 16:44:32.673965 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:32.673916    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk" podStartSLOduration=1.673899589 podStartE2EDuration="1.673899589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:44:32.671476722 +0000 UTC m=+3368.585533270" watchObservedRunningTime="2026-07-27 16:44:32.673899589 +0000 UTC m=+3368.587956137"
Jul 27 16:44:33.657197 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:33.657164    2576 generic.go:356] "Generic (PLEG): container finished" podID="21a2fb75-449d-481c-8968-2c81df36a40d" containerID="1e8fb533d61fe40667a088fcba244d9d58713e3fb900ee9b30aabf779d281aaa" exitCode=0
Jul 27 16:44:33.657672 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:33.657250    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk" event={"ID":"21a2fb75-449d-481c-8968-2c81df36a40d","Type":"ContainerDied","Data":"1e8fb533d61fe40667a088fcba244d9d58713e3fb900ee9b30aabf779d281aaa"}
Jul 27 16:44:34.797028 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:34.797003    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:34.928894 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:34.928814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca\") pod \"21a2fb75-449d-481c-8968-2c81df36a40d\" (UID: \"21a2fb75-449d-481c-8968-2c81df36a40d\") "
Jul 27 16:44:34.929169 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:34.929148    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21a2fb75-449d-481c-8968-2c81df36a40d" (UID: "21a2fb75-449d-481c-8968-2c81df36a40d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:44:35.030145 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:35.030085    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a2fb75-449d-481c-8968-2c81df36a40d-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:35.669237 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:35.669202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk" event={"ID":"21a2fb75-449d-481c-8968-2c81df36a40d","Type":"ContainerDied","Data":"4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee"}
Jul 27 16:44:35.669411 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:35.669249    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cfd8f7ebaafedf32564001042e77e152d62f8cd7cc21d4606af66cc45b6b5ee"
Jul 27 16:44:35.669411 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:35.669219    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"
Jul 27 16:44:42.083998 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.083959    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:44:42.084400 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.084243    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" podUID="4d301137-a9f8-45c2-8919-ab90391fd737" containerName="maas-api" containerID="cri-o://d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388" gracePeriod=30
Jul 27 16:44:42.342721 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.342654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:44:42.497124 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.497065    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca\") pod \"4d301137-a9f8-45c2-8919-ab90391fd737\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") "
Jul 27 16:44:42.497289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.497148    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") pod \"4d301137-a9f8-45c2-8919-ab90391fd737\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") "
Jul 27 16:44:42.497289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.497182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hh92\" (UniqueName: \"kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92\") pod \"4d301137-a9f8-45c2-8919-ab90391fd737\" (UID: \"4d301137-a9f8-45c2-8919-ab90391fd737\") "
Jul 27 16:44:42.497422 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.497398    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d301137-a9f8-45c2-8919-ab90391fd737" (UID: "4d301137-a9f8-45c2-8919-ab90391fd737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:44:42.499348 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.499325    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d301137-a9f8-45c2-8919-ab90391fd737" (UID: "4d301137-a9f8-45c2-8919-ab90391fd737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:44:42.499415 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.499348    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92" (OuterVolumeSpecName: "kube-api-access-4hh92") pod "4d301137-a9f8-45c2-8919-ab90391fd737" (UID: "4d301137-a9f8-45c2-8919-ab90391fd737"). InnerVolumeSpecName "kube-api-access-4hh92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:44:42.598712 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.598634    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d301137-a9f8-45c2-8919-ab90391fd737-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:42.598712 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.598670    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hh92\" (UniqueName: \"kubernetes.io/projected/4d301137-a9f8-45c2-8919-ab90391fd737-kube-api-access-4hh92\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:42.598712 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.598680    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d301137-a9f8-45c2-8919-ab90391fd737-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:42.702819 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.702790    2576 generic.go:356] "Generic (PLEG): container finished" podID="4d301137-a9f8-45c2-8919-ab90391fd737" containerID="d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388" exitCode=0
Jul 27 16:44:42.702982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.702840    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"
Jul 27 16:44:42.702982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.702871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" event={"ID":"4d301137-a9f8-45c2-8919-ab90391fd737","Type":"ContainerDied","Data":"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"}
Jul 27 16:44:42.702982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.702922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl" event={"ID":"4d301137-a9f8-45c2-8919-ab90391fd737","Type":"ContainerDied","Data":"4f0dfd080186369931e9a96b798ded78a1b421578301044b7c160e3e6d550db4"}
Jul 27 16:44:42.702982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.702945    2576 scope.go:117] "RemoveContainer" containerID="d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"
Jul 27 16:44:42.713383 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.713342    2576 scope.go:117] "RemoveContainer" containerID="d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"
Jul 27 16:44:42.713924 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:44:42.713901    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388\": container with ID starting with d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388 not found: ID does not exist" containerID="d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"
Jul 27 16:44:42.714013 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.713934    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388"} err="failed to get container status \"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388\": rpc error: code = NotFound desc = could not find container \"d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388\": container with ID starting with d8ddf83e3e158eee48aa6c34b448d1ee9fd4d3827fdbc67444994c0128ba2388 not found: ID does not exist"
Jul 27 16:44:42.720489 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.720467    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:44:42.723819 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:42.723801    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d32cda-75f687cb79-kb9dl"]
Jul 27 16:44:44.652848 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:44.652814    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d301137-a9f8-45c2-8919-ab90391fd737" path="/var/lib/kubelet/pods/4d301137-a9f8-45c2-8919-ab90391fd737/volumes"
Jul 27 16:44:45.593604 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:45.593573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:44:46.517574 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:46.517540    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"]
Jul 27 16:44:46.520476 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:46.520447    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d32cda-86gzk"]
Jul 27 16:44:46.652439 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:46.652412    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a2fb75-449d-481c-8968-2c81df36a40d" path="/var/lib/kubelet/pods/21a2fb75-449d-481c-8968-2c81df36a40d/volumes"
Jul 27 16:44:53.480377 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480342    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"]
Jul 27 16:44:53.480767 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480697    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d301137-a9f8-45c2-8919-ab90391fd737" containerName="maas-api"
Jul 27 16:44:53.480767 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480710    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d301137-a9f8-45c2-8919-ab90391fd737" containerName="maas-api"
Jul 27 16:44:53.480767 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480725    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a2fb75-449d-481c-8968-2c81df36a40d" containerName="revoke-keys"
Jul 27 16:44:53.480767 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480731    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a2fb75-449d-481c-8968-2c81df36a40d" containerName="revoke-keys"
Jul 27 16:44:53.480893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480792    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d301137-a9f8-45c2-8919-ab90391fd737" containerName="maas-api"
Jul 27 16:44:53.480893 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.480805    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a2fb75-449d-481c-8968-2c81df36a40d" containerName="revoke-keys"
Jul 27 16:44:53.485173 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.485153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:53.488064 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.488045    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:44:53.495378 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.495357    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"]
Jul 27 16:44:53.593708 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.593673    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-932843-hspgq\" (UID: \"02350ca0-1bd2-4f85-be0c-c86fb79747c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:53.695058 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.695021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-932843-hspgq\" (UID: \"02350ca0-1bd2-4f85-be0c-c86fb79747c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:53.695754 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.695730    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-932843-hspgq\" (UID: \"02350ca0-1bd2-4f85-be0c-c86fb79747c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:53.795669 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.795583    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:53.927587 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:53.927562    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"]
Jul 27 16:44:53.929839 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:44:53.929811    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02350ca0_1bd2_4f85_be0c_c86fb79747c3.slice/crio-746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3 WatchSource:0}: Error finding container 746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3: Status 404 returned error can't find the container with id 746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3
Jul 27 16:44:54.762491 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:54.762451    2576 generic.go:356] "Generic (PLEG): container finished" podID="02350ca0-1bd2-4f85-be0c-c86fb79747c3" containerID="bf84d60b4e4891ccb0349adb3c925b25a7d235496e06dc0e79b84a68c8b49309" exitCode=0
Jul 27 16:44:54.762864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:54.762510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq" event={"ID":"02350ca0-1bd2-4f85-be0c-c86fb79747c3","Type":"ContainerDied","Data":"bf84d60b4e4891ccb0349adb3c925b25a7d235496e06dc0e79b84a68c8b49309"}
Jul 27 16:44:54.762864 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:54.762563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq" event={"ID":"02350ca0-1bd2-4f85-be0c-c86fb79747c3","Type":"ContainerStarted","Data":"746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3"}
Jul 27 16:44:55.898096 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:55.898070    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:56.016408 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.016372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca\") pod \"02350ca0-1bd2-4f85-be0c-c86fb79747c3\" (UID: \"02350ca0-1bd2-4f85-be0c-c86fb79747c3\") "
Jul 27 16:44:56.016726 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.016702    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02350ca0-1bd2-4f85-be0c-c86fb79747c3" (UID: "02350ca0-1bd2-4f85-be0c-c86fb79747c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:44:56.117904 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.117833    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02350ca0-1bd2-4f85-be0c-c86fb79747c3-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:44:56.773216 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.773188    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"
Jul 27 16:44:56.773374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.773241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq" event={"ID":"02350ca0-1bd2-4f85-be0c-c86fb79747c3","Type":"ContainerDied","Data":"746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3"}
Jul 27 16:44:56.773374 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:44:56.773284    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="746fc34d2cb9a4c9473bff7da0605c2da2119ce69dd61380f9de54b6543134b3"
Jul 27 16:45:00.135645 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.135610    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"]
Jul 27 16:45:00.136020 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.135974    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02350ca0-1bd2-4f85-be0c-c86fb79747c3" containerName="revoke-keys"
Jul 27 16:45:00.136020 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.135988    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="02350ca0-1bd2-4f85-be0c-c86fb79747c3" containerName="revoke-keys"
Jul 27 16:45:00.136092 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.136079    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="02350ca0-1bd2-4f85-be0c-c86fb79747c3" containerName="revoke-keys"
Jul 27 16:45:00.139363 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.139341    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"]
Jul 27 16:45:00.139472 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.139435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"
Jul 27 16:45:00.142123 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.142088    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:45:00.142843 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.142826    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"]
Jul 27 16:45:00.142924 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.142911    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"
Jul 27 16:45:00.146044 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.146024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"
Jul 27 16:45:00.146465 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.146446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"]
Jul 27 16:45:00.149681 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.149659    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"]
Jul 27 16:45:00.153175 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.153152    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"]
Jul 27 16:45:00.153340 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.153322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"
Jul 27 16:45:00.162670 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.162644    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"
Jul 27 16:45:00.195400 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.195375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"
Jul 27 16:45:00.335426 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.335402    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"]
Jul 27 16:45:00.337891 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:45:00.337863    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda773bc71_ac86_4040_b210_05831552c195.slice/crio-69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce WatchSource:0}: Error finding container 69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce: Status 404 returned error can't find the container with id 69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce
Jul 27 16:45:00.389937 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.389915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"]
Jul 27 16:45:00.391737 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:45:00.391712    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ecb4188_c5b5_4484_9025_78c7de25cd2a.slice/crio-bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d WatchSource:0}: Error finding container bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d: Status 404 returned error can't find the container with id bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d
Jul 27 16:45:00.563904 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.563882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"]
Jul 27 16:45:00.565957 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:45:00.565926    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f48243_56e0_4313_a233_aa279588c859.slice/crio-7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0 WatchSource:0}: Error finding container 7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0: Status 404 returned error can't find the container with id 7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0
Jul 27 16:45:00.793231 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.793198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf" event={"ID":"4ecb4188-c5b5-4484-9025-78c7de25cd2a","Type":"ContainerStarted","Data":"0eb03ab7dcaed953536c581cf8584947d9716f1d62e991193b995ec83363a4b1"}
Jul 27 16:45:00.793357 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.793258    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf" event={"ID":"4ecb4188-c5b5-4484-9025-78c7de25cd2a","Type":"ContainerStarted","Data":"bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d"}
Jul 27 16:45:00.794887 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.794860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p" event={"ID":"87f48243-56e0-4313-a233-aa279588c859","Type":"ContainerStarted","Data":"e69acbdb213c4340ff08f6685149af0ed6721483a9321a0bea2925333421e19b"}
Jul 27 16:45:00.795016 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.794914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p" event={"ID":"87f48243-56e0-4313-a233-aa279588c859","Type":"ContainerStarted","Data":"7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0"}
Jul 27 16:45:00.796532 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.796502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f" event={"ID":"a773bc71-ac86-4040-b210-05831552c195","Type":"ContainerStarted","Data":"26583b30e1c48ee5dfbd8a37d3c1747f1fddff313d60d9c854b500006491b359"}
Jul 27 16:45:00.796644 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.796554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f" event={"ID":"a773bc71-ac86-4040-b210-05831552c195","Type":"ContainerStarted","Data":"69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce"}
Jul 27 16:45:00.809951 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.809908    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf" podStartSLOduration=0.809895698 podStartE2EDuration="809.895698ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:00.808812522 +0000 UTC m=+3396.722869072" watchObservedRunningTime="2026-07-27 16:45:00.809895698 +0000 UTC m=+3396.723952246"
Jul 27 16:45:00.824215 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.824172    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p" podStartSLOduration=0.824159586 podStartE2EDuration="824.159586ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:00.822645196 +0000 UTC m=+3396.736701745" watchObservedRunningTime="2026-07-27 16:45:00.824159586 +0000 UTC m=+3396.738216133"
Jul 27 16:45:00.837379 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:00.837338    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f" podStartSLOduration=0.837324825 podStartE2EDuration="837.324825ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:00.836920389 +0000 UTC m=+3396.750976938" watchObservedRunningTime="2026-07-27 16:45:00.837324825 +0000 UTC m=+3396.751381372"
Jul 27 16:45:01.803229 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.803096    2576 generic.go:356] "Generic (PLEG): container finished" podID="4ecb4188-c5b5-4484-9025-78c7de25cd2a" containerID="0eb03ab7dcaed953536c581cf8584947d9716f1d62e991193b995ec83363a4b1" exitCode=0
Jul 27 16:45:01.803229 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.803206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf" event={"ID":"4ecb4188-c5b5-4484-9025-78c7de25cd2a","Type":"ContainerDied","Data":"0eb03ab7dcaed953536c581cf8584947d9716f1d62e991193b995ec83363a4b1"}
Jul 27 16:45:01.804712 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.804686    2576 generic.go:356] "Generic (PLEG): container finished" podID="87f48243-56e0-4313-a233-aa279588c859" containerID="e69acbdb213c4340ff08f6685149af0ed6721483a9321a0bea2925333421e19b" exitCode=0
Jul 27 16:45:01.804839 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.804770    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p" event={"ID":"87f48243-56e0-4313-a233-aa279588c859","Type":"ContainerDied","Data":"e69acbdb213c4340ff08f6685149af0ed6721483a9321a0bea2925333421e19b"}
Jul 27 16:45:01.806217 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.806199    2576 generic.go:356] "Generic (PLEG): container finished" podID="a773bc71-ac86-4040-b210-05831552c195" containerID="26583b30e1c48ee5dfbd8a37d3c1747f1fddff313d60d9c854b500006491b359" exitCode=0
Jul 27 16:45:01.806289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:01.806225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f" event={"ID":"a773bc71-ac86-4040-b210-05831552c195","Type":"ContainerDied","Data":"26583b30e1c48ee5dfbd8a37d3c1747f1fddff313d60d9c854b500006491b359"}
Jul 27 16:45:02.944581 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:02.944554    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"
Jul 27 16:45:02.967065 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:02.967037    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"
Jul 27 16:45:02.993216 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:02.993190    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"
Jul 27 16:45:03.817038 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.817006    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"
Jul 27 16:45:03.817232 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.817008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf" event={"ID":"4ecb4188-c5b5-4484-9025-78c7de25cd2a","Type":"ContainerDied","Data":"bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d"}
Jul 27 16:45:03.817232 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.817146    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9e9f56809b2a372bf45c28464d441a9717a066a299901532c54000568cd21d"
Jul 27 16:45:03.818307 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.818279    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"
Jul 27 16:45:03.818449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.818308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p" event={"ID":"87f48243-56e0-4313-a233-aa279588c859","Type":"ContainerDied","Data":"7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0"}
Jul 27 16:45:03.818449 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.818352    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe82b2ed6f30eaffcef01740796a2c1afa3150c71ce09cb3c0df4e6b2f52cd0"
Jul 27 16:45:03.819565 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.819548    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f"
Jul 27 16:45:03.819565 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.819554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29752845-h4t2f" event={"ID":"a773bc71-ac86-4040-b210-05831552c195","Type":"ContainerDied","Data":"69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce"}
Jul 27 16:45:03.819702 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.819595    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e33807c4d9805914da30446acc90f9e23158ba14a9b7a29173a89a5c53e8ce"
Jul 27 16:45:03.979140 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.979094    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:45:03.979832 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:03.979802    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" podUID="85a0c2ca-8181-4cf8-9a0b-841360413360" containerName="maas-api" containerID="cri-o://957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a" gracePeriod=30
Jul 27 16:45:04.005944 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.005916    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"]
Jul 27 16:45:04.009438 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.009417    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-932843-29752845-pzkkf"]
Jul 27 16:45:04.231899 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.231872    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:45:04.294552 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.294522    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca\") pod \"85a0c2ca-8181-4cf8-9a0b-841360413360\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") "
Jul 27 16:45:04.294684 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.294615    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v\") pod \"85a0c2ca-8181-4cf8-9a0b-841360413360\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") "
Jul 27 16:45:04.294684 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.294661    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls\") pod \"85a0c2ca-8181-4cf8-9a0b-841360413360\" (UID: \"85a0c2ca-8181-4cf8-9a0b-841360413360\") "
Jul 27 16:45:04.294861 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.294840    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85a0c2ca-8181-4cf8-9a0b-841360413360" (UID: "85a0c2ca-8181-4cf8-9a0b-841360413360"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:45:04.296646 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.296616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85a0c2ca-8181-4cf8-9a0b-841360413360" (UID: "85a0c2ca-8181-4cf8-9a0b-841360413360"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:45:04.296758 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.296681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v" (OuterVolumeSpecName: "kube-api-access-f7w2v") pod "85a0c2ca-8181-4cf8-9a0b-841360413360" (UID: "85a0c2ca-8181-4cf8-9a0b-841360413360"). InnerVolumeSpecName "kube-api-access-f7w2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:45:04.395916 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.395841    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7w2v\" (UniqueName: \"kubernetes.io/projected/85a0c2ca-8181-4cf8-9a0b-841360413360-kube-api-access-f7w2v\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:04.395916 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.395870    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85a0c2ca-8181-4cf8-9a0b-841360413360-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:04.395916 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.395880    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85a0c2ca-8181-4cf8-9a0b-841360413360-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:04.652016 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.651935    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ecb4188-c5b5-4484-9025-78c7de25cd2a" path="/var/lib/kubelet/pods/4ecb4188-c5b5-4484-9025-78c7de25cd2a/volumes"
Jul 27 16:45:04.826013 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.825974    2576 generic.go:356] "Generic (PLEG): container finished" podID="85a0c2ca-8181-4cf8-9a0b-841360413360" containerID="957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a" exitCode=0
Jul 27 16:45:04.826189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.826023    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"
Jul 27 16:45:04.826189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.826063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" event={"ID":"85a0c2ca-8181-4cf8-9a0b-841360413360","Type":"ContainerDied","Data":"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"}
Jul 27 16:45:04.826189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.826145    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2" event={"ID":"85a0c2ca-8181-4cf8-9a0b-841360413360","Type":"ContainerDied","Data":"bb70b7426826ca26786b6fcc8ed62104277a0faaeea964c42671368a7df130d2"}
Jul 27 16:45:04.826189 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.826176    2576 scope.go:117] "RemoveContainer" containerID="957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"
Jul 27 16:45:04.839721 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.839705    2576 scope.go:117] "RemoveContainer" containerID="957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"
Jul 27 16:45:04.839979 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:45:04.839959    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a\": container with ID starting with 957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a not found: ID does not exist" containerID="957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"
Jul 27 16:45:04.840053 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.839985    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a"} err="failed to get container status \"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a\": rpc error: code = NotFound desc = could not find container \"957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a\": container with ID starting with 957c66a221309ad3ede5be2faf1a396a604fb1f935169a040e5bc13dd82e679a not found: ID does not exist"
Jul 27 16:45:04.844152 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.844128    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:45:04.852351 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:04.852328    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-932843-c95667cb6-k6md2"]
Jul 27 16:45:06.652918 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:06.652884    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85a0c2ca-8181-4cf8-9a0b-841360413360" path="/var/lib/kubelet/pods/85a0c2ca-8181-4cf8-9a0b-841360413360/volumes"
Jul 27 16:45:07.392757 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:07.392724    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:45:08.549089 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:08.549043    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"]
Jul 27 16:45:08.553144 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:08.553114    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-932843-hspgq"]
Jul 27 16:45:08.653908 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:08.653874    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02350ca0-1bd2-4f85-be0c-c86fb79747c3" path="/var/lib/kubelet/pods/02350ca0-1bd2-4f85-be0c-c86fb79747c3/volumes"
Jul 27 16:45:15.796540 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796505    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"]
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796834    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87f48243-56e0-4313-a233-aa279588c859" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796844    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="87f48243-56e0-4313-a233-aa279588c859" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796858    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ecb4188-c5b5-4484-9025-78c7de25cd2a" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796863    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ecb4188-c5b5-4484-9025-78c7de25cd2a" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796873    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85a0c2ca-8181-4cf8-9a0b-841360413360" containerName="maas-api"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796879    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="85a0c2ca-8181-4cf8-9a0b-841360413360" containerName="maas-api"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796885    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a773bc71-ac86-4040-b210-05831552c195" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a773bc71-ac86-4040-b210-05831552c195" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796936    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a773bc71-ac86-4040-b210-05831552c195" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796944    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ecb4188-c5b5-4484-9025-78c7de25cd2a" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796953    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="87f48243-56e0-4313-a233-aa279588c859" containerName="cleanup"
Jul 27 16:45:15.798715 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.796960    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="85a0c2ca-8181-4cf8-9a0b-841360413360" containerName="maas-api"
Jul 27 16:45:15.799845 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.799829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:15.802587 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.802568    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqs6d\""
Jul 27 16:45:15.808085 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.808058    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"]
Jul 27 16:45:15.896041 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.896009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9\" (UID: \"bde74103-49f9-48cd-9e41-d5048bf20f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:15.997017 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.996988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9\" (UID: \"bde74103-49f9-48cd-9e41-d5048bf20f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:15.997676 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:15.997655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9\" (UID: \"bde74103-49f9-48cd-9e41-d5048bf20f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:16.110917 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:16.110817    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:16.452258 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:16.452232    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"]
Jul 27 16:45:16.454282 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:45:16.454257    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbde74103_49f9_48cd_9e41_d5048bf20f9b.slice/crio-22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f WatchSource:0}: Error finding container 22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f: Status 404 returned error can't find the container with id 22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f
Jul 27 16:45:16.888769 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:16.888674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9" event={"ID":"bde74103-49f9-48cd-9e41-d5048bf20f9b","Type":"ContainerStarted","Data":"35eabdd1bcb5d64a59a56cd4f47903a692225ab0972e13ccb13b60b17ca38794"}
Jul 27 16:45:16.889149 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:16.888775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9" event={"ID":"bde74103-49f9-48cd-9e41-d5048bf20f9b","Type":"ContainerStarted","Data":"22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f"}
Jul 27 16:45:16.908388 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:16.908344    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9" podStartSLOduration=1.908327917 podStartE2EDuration="1.908327917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:16.906449997 +0000 UTC m=+3412.820506544" watchObservedRunningTime="2026-07-27 16:45:16.908327917 +0000 UTC m=+3412.822384478"
Jul 27 16:45:17.894691 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:17.894653    2576 generic.go:356] "Generic (PLEG): container finished" podID="bde74103-49f9-48cd-9e41-d5048bf20f9b" containerID="35eabdd1bcb5d64a59a56cd4f47903a692225ab0972e13ccb13b60b17ca38794" exitCode=0
Jul 27 16:45:17.895077 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:17.894726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9" event={"ID":"bde74103-49f9-48cd-9e41-d5048bf20f9b","Type":"ContainerDied","Data":"35eabdd1bcb5d64a59a56cd4f47903a692225ab0972e13ccb13b60b17ca38794"}
Jul 27 16:45:19.034703 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.034678    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:19.124453 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.124425    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca\") pod \"bde74103-49f9-48cd-9e41-d5048bf20f9b\" (UID: \"bde74103-49f9-48cd-9e41-d5048bf20f9b\") "
Jul 27 16:45:19.124796 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.124771    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bde74103-49f9-48cd-9e41-d5048bf20f9b" (UID: "bde74103-49f9-48cd-9e41-d5048bf20f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:45:19.225821 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.225790    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde74103-49f9-48cd-9e41-d5048bf20f9b-maas-api-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:19.906560 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.906534    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"
Jul 27 16:45:19.906728 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.906579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9" event={"ID":"bde74103-49f9-48cd-9e41-d5048bf20f9b","Type":"ContainerDied","Data":"22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f"}
Jul 27 16:45:19.906728 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:19.906639    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22f55a49e8e820554d3425aab9683f6469bfef5c7013ec1247f62ed139fba14f"
Jul 27 16:45:26.349860 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.349813    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:45:26.350982 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.350701    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" podUID="652d348b-10f7-4326-978f-abacab8dc9a5" containerName="maas-api" containerID="cri-o://43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6" gracePeriod=30
Jul 27 16:45:26.376920 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.376892    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"]
Jul 27 16:45:26.380391 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.380358    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-1f21d6-29752845-8k98p"]
Jul 27 16:45:26.602650 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.602595    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:45:26.652909 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.652877    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87f48243-56e0-4313-a233-aa279588c859" path="/var/lib/kubelet/pods/87f48243-56e0-4313-a233-aa279588c859/volumes"
Jul 27 16:45:26.690234 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.690211    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tts8\" (UniqueName: \"kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8\") pod \"652d348b-10f7-4326-978f-abacab8dc9a5\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") "
Jul 27 16:45:26.690347 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.690265    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") pod \"652d348b-10f7-4326-978f-abacab8dc9a5\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") "
Jul 27 16:45:26.690406 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.690355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca\") pod \"652d348b-10f7-4326-978f-abacab8dc9a5\" (UID: \"652d348b-10f7-4326-978f-abacab8dc9a5\") "
Jul 27 16:45:26.690776 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.690731    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "652d348b-10f7-4326-978f-abacab8dc9a5" (UID: "652d348b-10f7-4326-978f-abacab8dc9a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 16:45:26.692628 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.692598    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8" (OuterVolumeSpecName: "kube-api-access-9tts8") pod "652d348b-10f7-4326-978f-abacab8dc9a5" (UID: "652d348b-10f7-4326-978f-abacab8dc9a5"). InnerVolumeSpecName "kube-api-access-9tts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 16:45:26.692754 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.692693    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "652d348b-10f7-4326-978f-abacab8dc9a5" (UID: "652d348b-10f7-4326-978f-abacab8dc9a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 16:45:26.792117 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.792068    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/652d348b-10f7-4326-978f-abacab8dc9a5-openshift-service-ca\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:26.792117 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.792095    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tts8\" (UniqueName: \"kubernetes.io/projected/652d348b-10f7-4326-978f-abacab8dc9a5-kube-api-access-9tts8\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:26.792356 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.792132    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/652d348b-10f7-4326-978f-abacab8dc9a5-maas-api-tls\") on node \"ip-10-0-135-171.ec2.internal\" DevicePath \"\""
Jul 27 16:45:26.943130 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.943083    2576 generic.go:356] "Generic (PLEG): container finished" podID="652d348b-10f7-4326-978f-abacab8dc9a5" containerID="43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6" exitCode=0
Jul 27 16:45:26.943267 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.943140    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"
Jul 27 16:45:26.943267 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.943166    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" event={"ID":"652d348b-10f7-4326-978f-abacab8dc9a5","Type":"ContainerDied","Data":"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"}
Jul 27 16:45:26.943267 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.943225    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h" event={"ID":"652d348b-10f7-4326-978f-abacab8dc9a5","Type":"ContainerDied","Data":"3c1f75d61c72fa4ebe2c5dd71762bca65f6317f2b36b0bcb97bd0ac144612556"}
Jul 27 16:45:26.943267 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.943248    2576 scope.go:117] "RemoveContainer" containerID="43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"
Jul 27 16:45:26.952383 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.952355    2576 scope.go:117] "RemoveContainer" containerID="43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"
Jul 27 16:45:26.952657 ip-10-0-135-171 kubenswrapper[2576]: E0727 16:45:26.952628    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6\": container with ID starting with 43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6 not found: ID does not exist" containerID="43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"
Jul 27 16:45:26.952778 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.952665    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6"} err="failed to get container status \"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6\": rpc error: code = NotFound desc = could not find container \"43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6\": container with ID starting with 43098d4603b69bced32f35e33d4e801c870903e2f988ffd73cdb90ea818625f6 not found: ID does not exist"
Jul 27 16:45:26.975759 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.975737    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:45:26.982732 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:26.982713    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1f21d6-68495dcfc8-slz7h"]
Jul 27 16:45:28.652409 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:28.652379    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="652d348b-10f7-4326-978f-abacab8dc9a5" path="/var/lib/kubelet/pods/652d348b-10f7-4326-978f-abacab8dc9a5/volumes"
Jul 27 16:45:29.405201 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:29.405160    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-r2v9s"]
Jul 27 16:45:30.898947 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:30.898915    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"]
Jul 27 16:45:30.907289 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:30.907263    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1f21d6-r29g9"]
Jul 27 16:45:32.653613 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:32.653578    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bde74103-49f9-48cd-9e41-d5048bf20f9b" path="/var/lib/kubelet/pods/bde74103-49f9-48cd-9e41-d5048bf20f9b/volumes"
Jul 27 16:45:44.029746 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:44.029714    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z88l8_19751475-dae3-4ddd-acec-aec1ec215ed8/manager/0.log"
Jul 27 16:45:44.619665 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:44.619632    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc7975cbd-89j4t_07678090-99f6-49cc-8400-9e55b8a6ac4d/copy-manifests/0.log"
Jul 27 16:45:44.638467 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:44.638431    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc7975cbd-89j4t_07678090-99f6-49cc-8400-9e55b8a6ac4d/manager/0.log"
Jul 27 16:45:45.386610 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:45.386577    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752830-w4vzm_aa95cee2-89e5-415f-899f-612dba8d1a98/cleanup/0.log"
Jul 27 16:45:45.498030 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:45.497997    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29752845-h4t2f_a773bc71-ac86-4040-b210-05831552c195/cleanup/0.log"
Jul 27 16:45:47.115901 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:47.115870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vvpmn_d9ad7eb2-85e0-48df-be58-34c511c63589/manager/0.log"
Jul 27 16:45:47.494354 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:47.494323    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sjb6h_d7a38a8c-b747-4d09-97a8-ed591718d250/registry-server/0.log"
Jul 27 16:45:47.646315 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:47.646287    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qq5p2_46b86168-5e30-4219-93be-d35e6abb553a/manager/0.log"
Jul 27 16:45:47.758389 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:47.758301    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-r2v9s_e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c/limitador/0.log"
Jul 27 16:45:49.142487 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:49.142459    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6848cdb7df-qbb29_395f36f7-8a82-499c-b1e8-f698f52f2594/router/0.log"
Jul 27 16:45:49.889137 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:49.889094    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2927r_36d637e8-fb73-4129-b094-b402be7cb32b/main/0.log"
Jul 27 16:45:50.124717 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:50.124666    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-mh7qp_4b7c0030-5c11-4f4b-a30e-3497fe25a6b4/main/0.log"
Jul 27 16:45:55.707225 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707193    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6nds/must-gather-pbld4"]
Jul 27 16:45:55.707590 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707562    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bde74103-49f9-48cd-9e41-d5048bf20f9b" containerName="revoke-keys"
Jul 27 16:45:55.707590 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707577    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bde74103-49f9-48cd-9e41-d5048bf20f9b" containerName="revoke-keys"
Jul 27 16:45:55.707590 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707586    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="652d348b-10f7-4326-978f-abacab8dc9a5" containerName="maas-api"
Jul 27 16:45:55.707590 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707592    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="652d348b-10f7-4326-978f-abacab8dc9a5" containerName="maas-api"
Jul 27 16:45:55.707719 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707662    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bde74103-49f9-48cd-9e41-d5048bf20f9b" containerName="revoke-keys"
Jul 27 16:45:55.707719 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.707671    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="652d348b-10f7-4326-978f-abacab8dc9a5" containerName="maas-api"
Jul 27 16:45:55.710643 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.710626    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.713265 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.713238    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6nds\"/\"kube-root-ca.crt\""
Jul 27 16:45:55.713400 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.713291    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t6nds\"/\"openshift-service-ca.crt\""
Jul 27 16:45:55.723662 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.723643    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6nds/must-gather-pbld4"]
Jul 27 16:45:55.834408 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.834381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/01452abb-ebb4-4021-8471-a937bf2e43a4-must-gather-output\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.834558 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.834412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nctk5\" (UniqueName: \"kubernetes.io/projected/01452abb-ebb4-4021-8471-a937bf2e43a4-kube-api-access-nctk5\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.935368 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.935339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/01452abb-ebb4-4021-8471-a937bf2e43a4-must-gather-output\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.935368 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.935368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nctk5\" (UniqueName: \"kubernetes.io/projected/01452abb-ebb4-4021-8471-a937bf2e43a4-kube-api-access-nctk5\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.935701 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.935678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/01452abb-ebb4-4021-8471-a937bf2e43a4-must-gather-output\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:55.945161 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:55.945134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nctk5\" (UniqueName: \"kubernetes.io/projected/01452abb-ebb4-4021-8471-a937bf2e43a4-kube-api-access-nctk5\") pod \"must-gather-pbld4\" (UID: \"01452abb-ebb4-4021-8471-a937bf2e43a4\") " pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:56.020277 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:56.020173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6nds/must-gather-pbld4"
Jul 27 16:45:56.156565 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:56.156539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6nds/must-gather-pbld4"]
Jul 27 16:45:56.158960 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:45:56.158935    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01452abb_ebb4_4021_8471_a937bf2e43a4.slice/crio-70246a5b484aab7e2d7128dd7af3ccc3d7d09e8c3d410738e7f4d0e50aa68688 WatchSource:0}: Error finding container 70246a5b484aab7e2d7128dd7af3ccc3d7d09e8c3d410738e7f4d0e50aa68688: Status 404 returned error can't find the container with id 70246a5b484aab7e2d7128dd7af3ccc3d7d09e8c3d410738e7f4d0e50aa68688
Jul 27 16:45:57.098182 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:57.098134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6nds/must-gather-pbld4" event={"ID":"01452abb-ebb4-4021-8471-a937bf2e43a4","Type":"ContainerStarted","Data":"70246a5b484aab7e2d7128dd7af3ccc3d7d09e8c3d410738e7f4d0e50aa68688"}
Jul 27 16:45:58.106738 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:58.106703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6nds/must-gather-pbld4" event={"ID":"01452abb-ebb4-4021-8471-a937bf2e43a4","Type":"ContainerStarted","Data":"cf15545bb619affd6df8822936d799f49fb03220a8c2cf0a6e03d99a7500d33c"}
Jul 27 16:45:58.106738 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:58.106746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6nds/must-gather-pbld4" event={"ID":"01452abb-ebb4-4021-8471-a937bf2e43a4","Type":"ContainerStarted","Data":"5e51b714bafa2e30278f74aef9fcf58daf2e9056c28f3f5a37b18df760f2d9a9"}
Jul 27 16:45:58.124413 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:58.124365    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6nds/must-gather-pbld4" podStartSLOduration=1.65818512 podStartE2EDuration="3.124349527s" totalImagesPullingTime="1.466164407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 16:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:45:58.122524917 +0000 UTC m=+3454.036581490" watchObservedRunningTime="2026-07-27 16:45:58.124349527 +0000 UTC m=+3454.038406074"
Jul 27 16:45:59.316666 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:59.316620    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qd4fh_f8a436aa-e355-41f9-ae1b-8caa9b42dcf8/global-pull-secret-syncer/0.log"
Jul 27 16:45:59.441193 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:59.441141    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dp29f_dd0534f6-4899-4330-a551-9c446059d5e8/konnectivity-agent/0.log"
Jul 27 16:45:59.469150 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:45:59.469081    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-171.ec2.internal_9a03fbe185acdf451816f7e23b5f888d/haproxy/0.log"
Jul 27 16:46:04.514341 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:04.514307    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vvpmn_d9ad7eb2-85e0-48df-be58-34c511c63589/manager/0.log"
Jul 27 16:46:04.626628 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:04.626300    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-sjb6h_d7a38a8c-b747-4d09-97a8-ed591718d250/registry-server/0.log"
Jul 27 16:46:04.898858 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:04.898824    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-qq5p2_46b86168-5e30-4219-93be-d35e6abb553a/manager/0.log"
Jul 27 16:46:04.925990 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:04.925806    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-r2v9s_e2c6fb1d-eb12-4cd3-982b-3a36b2982f4c/limitador/0.log"
Jul 27 16:46:06.734851 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:06.734772    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5d59984c4f-2znxd_a61d0294-8692-473c-891f-4bcb78776919/metrics-server/0.log"
Jul 27 16:46:06.909557 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:06.909526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm5ch_aae884ec-8128-463f-bf0f-dbd4545b3712/node-exporter/0.log"
Jul 27 16:46:06.939737 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:06.939706    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm5ch_aae884ec-8128-463f-bf0f-dbd4545b3712/kube-rbac-proxy/0.log"
Jul 27 16:46:06.966044 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:06.966015    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm5ch_aae884ec-8128-463f-bf0f-dbd4545b3712/init-textfile/0.log"
Jul 27 16:46:07.193442 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.193412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/prometheus/0.log"
Jul 27 16:46:07.221513 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.221484    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/config-reloader/0.log"
Jul 27 16:46:07.245401 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.245378    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/thanos-sidecar/0.log"
Jul 27 16:46:07.277093 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.277066    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/kube-rbac-proxy-web/0.log"
Jul 27 16:46:07.300270 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.300238    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/kube-rbac-proxy/0.log"
Jul 27 16:46:07.325131 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.325086    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/kube-rbac-proxy-thanos/0.log"
Jul 27 16:46:07.349140 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:07.349080    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_eef6e47d-7d51-4068-bbde-1f558a5612f6/init-config-reloader/0.log"
Jul 27 16:46:08.896837 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:08.896805    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-xnnhk_3ac1db04-c9e6-40e4-afda-4f80fb4ad792/networking-console-plugin/0.log"
Jul 27 16:46:09.466378 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.466351    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/2.log"
Jul 27 16:46:09.471062 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.471033    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qq9gn_98e29607-f7f5-4282-98d9-b2dd928e34d5/console-operator/3.log"
Jul 27 16:46:09.907739 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.907657    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"]
Jul 27 16:46:09.912334 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.912302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:09.916076 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.916057    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t6nds\"/\"default-dockercfg-zspp8\""
Jul 27 16:46:09.919978 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:09.919953    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"]
Jul 27 16:46:10.082829 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.082798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-proc\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.083018 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.082869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-sys\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.083018 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.082894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-podres\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.083018 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.082957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvzr\" (UniqueName: \"kubernetes.io/projected/1b81a520-92e4-40b3-927d-91be6eaf0b28-kube-api-access-6hvzr\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.083018 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.082983    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-lib-modules\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.183799 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183705    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-sys\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.183799 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-podres\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvzr\" (UniqueName: \"kubernetes.io/projected/1b81a520-92e4-40b3-927d-91be6eaf0b28-kube-api-access-6hvzr\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-lib-modules\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-sys\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-proc\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183960    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-podres\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184031 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.183995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-lib-modules\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.184330 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.184030    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1b81a520-92e4-40b3-927d-91be6eaf0b28-proc\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.194726 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.194589    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvzr\" (UniqueName: \"kubernetes.io/projected/1b81a520-92e4-40b3-927d-91be6eaf0b28-kube-api-access-6hvzr\") pod \"perf-node-gather-daemonset-dv86c\" (UID: \"1b81a520-92e4-40b3-927d-91be6eaf0b28\") " pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.227130 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.227060    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:10.398370 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.398342    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"]
Jul 27 16:46:10.401775 ip-10-0-135-171 kubenswrapper[2576]: W0727 16:46:10.401745    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1b81a520_92e4_40b3_927d_91be6eaf0b28.slice/crio-25ee6f81a95035e5c1ace9bd218fc3b81305beef4e2c07275057b6e8bb13211a WatchSource:0}: Error finding container 25ee6f81a95035e5c1ace9bd218fc3b81305beef4e2c07275057b6e8bb13211a: Status 404 returned error can't find the container with id 25ee6f81a95035e5c1ace9bd218fc3b81305beef4e2c07275057b6e8bb13211a
Jul 27 16:46:10.764546 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:10.764462    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-d65gl_ea31c5ca-0c82-4d56-b52a-011f3de07eaa/volume-data-source-validator/0.log"
Jul 27 16:46:11.206909 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.206872    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c" event={"ID":"1b81a520-92e4-40b3-927d-91be6eaf0b28","Type":"ContainerStarted","Data":"513a8ab5c81de57489d2b701f9373bf5f201d1b2669af4dd50b07700941ca91f"}
Jul 27 16:46:11.207348 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.206936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c" event={"ID":"1b81a520-92e4-40b3-927d-91be6eaf0b28","Type":"ContainerStarted","Data":"25ee6f81a95035e5c1ace9bd218fc3b81305beef4e2c07275057b6e8bb13211a"}
Jul 27 16:46:11.207348 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.206968    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:11.226503 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.226441    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c" podStartSLOduration=2.226419793 podStartE2EDuration="2.226419793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 16:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 16:46:11.225447959 +0000 UTC m=+3467.139504508" watchObservedRunningTime="2026-07-27 16:46:11.226419793 +0000 UTC m=+3467.140476346"
Jul 27 16:46:11.878054 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.878023    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxq7_dc31d968-2a9a-4cb6-8ad5-3b40b9453eac/dns/0.log"
Jul 27 16:46:11.912838 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:11.912792    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mbxq7_dc31d968-2a9a-4cb6-8ad5-3b40b9453eac/kube-rbac-proxy/0.log"
Jul 27 16:46:12.016700 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:12.016660    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kdn7c_a1da3b24-14ce-4aa1-96ff-b045734336da/dns-node-resolver/0.log"
Jul 27 16:46:12.602210 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:12.602174    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64849d8666-t8nvz_970c6b75-4ec1-49aa-93cc-8d59b99225a9/registry/0.log"
Jul 27 16:46:12.623306 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:12.623274    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64849d8666-t8nvz_970c6b75-4ec1-49aa-93cc-8d59b99225a9/registry/1.log"
Jul 27 16:46:12.680658 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:12.680627    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sst8w_73411ea9-8cbf-44ed-b28d-b1dba00cd762/node-ca/0.log"
Jul 27 16:46:14.152051 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:14.152014    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6848cdb7df-qbb29_395f36f7-8a82-499c-b1e8-f698f52f2594/router/0.log"
Jul 27 16:46:15.061662 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.061634    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-74bjz_9881139c-b0ec-4a87-8c39-73f5c23964a3/serve-healthcheck-canary/0.log"
Jul 27 16:46:15.690365 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.690335    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-h7n4w_a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa/insights-operator/0.log"
Jul 27 16:46:15.693441 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.693413    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-h7n4w_a4f5c35a-9ee3-4a33-a2f9-cb3fb9ae79fa/insights-operator/1.log"
Jul 27 16:46:15.836051 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.835998    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mv9h6_0b67406a-53cf-455e-ae4d-9329dce3d440/kube-rbac-proxy/0.log"
Jul 27 16:46:15.868651 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.868620    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mv9h6_0b67406a-53cf-455e-ae4d-9329dce3d440/exporter/0.log"
Jul 27 16:46:15.901576 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:15.901528    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mv9h6_0b67406a-53cf-455e-ae4d-9329dce3d440/extractor/0.log"
Jul 27 16:46:17.224483 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:17.224448    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t6nds/perf-node-gather-daemonset-dv86c"
Jul 27 16:46:18.529710 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:18.529683    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z88l8_19751475-dae3-4ddd-acec-aec1ec215ed8/manager/0.log"
Jul 27 16:46:18.986431 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:18.986393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc7975cbd-89j4t_07678090-99f6-49cc-8400-9e55b8a6ac4d/manager/0.log"
Jul 27 16:46:19.006823 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:19.006791    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7fc7975cbd-89j4t_07678090-99f6-49cc-8400-9e55b8a6ac4d/copy-manifests/0.log"
Jul 27 16:46:19.880509 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:19.880429    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6848cdb7df-qbb29_395f36f7-8a82-499c-b1e8-f698f52f2594/router/0.log"
Jul 27 16:46:20.378584 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:20.378558    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c7d956869-nszgh_2101e000-95f9-4d62-87de-f08b7b154a82/manager/0.log"
Jul 27 16:46:21.576555 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:21.576521    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6848cdb7df-qbb29_395f36f7-8a82-499c-b1e8-f698f52f2594/router/0.log"
Jul 27 16:46:27.018062 ip-10-0-135-171 kubenswrapper[2576]: I0727 16:46:27.018033    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mmk5r_5c391c2f-95c7-491f-800c-2071ad9d4887/kube-multus-additional-cni-plugins/0.log"