Jul 27 21:45:34.234797 ip-10-0-143-110 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 21:45:34.608084 ip-10-0-143-110 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:34.608084 ip-10-0-143-110 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 21:45:34.608084 ip-10-0-143-110 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:34.608084 ip-10-0-143-110 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 21:45:34.608084 ip-10-0-143-110 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 21:45:34.610388 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.610298    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 21:45:34.614539 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614523    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.614539 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614539    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614542    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614546    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614556    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614560    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614564    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614569    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614571    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614574    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614577    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614579    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614583    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614587    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614590    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614593    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614596    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614598    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614604    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614606    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.614615 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614609    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614611    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614614    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614617    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614620    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614622    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614626    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614629    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614632    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614634    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614636    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614639    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614641    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614644    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614646    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614649    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614653    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614655    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614658    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614660    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.615067 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614662    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614665    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614667    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614670    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614672    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614675    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614677    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614679    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614682    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614684    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614688    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614691    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614693    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614696    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614698    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614701    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614703    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614706    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614708    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614711    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.615805 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614713    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614716    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614718    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614722    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614725    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614728    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614731    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614733    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614736    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614739    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614742    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614745    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614748    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614750    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614753    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614755    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614758    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614760    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614762    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614765    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.616592 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614767    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614771    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614775    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614777    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614780    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614782    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614786    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614788    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614791    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614794    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614796    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614799    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614801    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614804    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614806    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614809    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614811    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614813    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.617076 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.614816    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.619228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619210    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.619228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619228    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619232    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619235    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619248    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619251    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619254    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619257    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619259    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619262    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619265    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619267    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619271    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619274    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619276    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619279    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619282    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619285    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619287    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619290    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619292    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.619293 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619296    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619299    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619302    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619305    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619307    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619311    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619316    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619319    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619322    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619325    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619332    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619335    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619338    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619340    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619344    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619349    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619352    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619355    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619357    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.619790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619360    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619362    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619366    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619368    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619371    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619373    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619376    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619378    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619381    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619383    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619386    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619389    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619391    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619394    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619397    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619400    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619403    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619405    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619408    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619410    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.620256 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619413    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619415    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619418    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619420    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619423    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619426    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619428    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619431    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619433    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619436    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619438    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619440    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619443    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619461    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619466    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619470    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619472    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619475    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619477    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619480    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.620756 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619482    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619485    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619488    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619492    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619496    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619499    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619502    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619505    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619508    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619510    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619514    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619518    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619521    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619524    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619526    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619529    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619532    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619534    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619538    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.621228 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619945    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619951    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619954    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619956    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619959    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619963    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619966    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619969    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619972    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619974    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619978    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619981    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619983    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619986    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619989    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619991    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619995    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.619997    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620000    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620003    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.621706 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620005    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620008    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620011    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620013    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620017    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620021    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620024    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620026    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620029    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620032    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620034    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620036    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620039    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620041    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620044    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620047    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620049    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620052    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620054    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.622200 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620057    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620059    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620062    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620064    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620067    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620069    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620072    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620074    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620077    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620079    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620083    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620087    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620091    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620093    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620096    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620099    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620101    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620104    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620106    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.622699 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620108    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620111    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620113    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620116    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620118    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620121    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620123    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620126    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620128    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620131    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620133    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620136    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620139    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620142    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620144    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620149    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620152    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620155    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620158    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.623156 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620161    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620163    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620166    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620168    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620171    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620174    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620177    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620180    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620183    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620185    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620188    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620190    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620193    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620195    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620198    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620200    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620203    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620205    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620208    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620210    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.623662 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620213    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.620215    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620295    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620305    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620312    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620316    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620320    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620323    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620328    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620332    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620335    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620338    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620341    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620345    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620350    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620355    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620360    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620365    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620370    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.620374    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621107    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621111    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621114    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621117    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 21:45:34.624139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621120    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621125    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621128    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621131    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621134    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621137    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621140    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621143    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621146    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621149    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621153    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621157    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621160    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621163    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621166    2560 flags.go:64] FLAG: --enable-server="true"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621169    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621174    2560 flags.go:64] FLAG: --event-burst="100"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621177    2560 flags.go:64] FLAG: --event-qps="50"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621180    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621183    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621186    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621189    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621193    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621196    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621198    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 27 21:45:34.624759 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621201    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621204    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621208    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621211    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621214    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621217    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621220    2560 flags.go:64] FLAG: --feature-gates=""
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621224    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621227    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621230    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621233    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621236    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621239    2560 flags.go:64] FLAG: --help="false"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621242    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-143-110.ec2.internal"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621245    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621248    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621251    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621254    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621258    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621261    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621263    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621266    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621269    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 21:45:34.625341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621272    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621275    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621278    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621281    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621283    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621287    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621289    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621292    2560 flags.go:64] FLAG: --lock-file=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621295    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621297    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621301    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621306    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621310    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621313    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621315    2560 flags.go:64] FLAG: --logging-format="text"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621318    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621322    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621324    2560 flags.go:64] FLAG: --manifest-url=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621327    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621332    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621335    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621339    2560 flags.go:64] FLAG: --max-pods="110"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621342    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621345    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621348    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 21:45:34.625927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621350    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621353    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621356    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621359    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621367    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621370    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621373    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621376    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621379    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621384    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621387    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621390    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621393    2560 flags.go:64] FLAG: --port="10250"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621396    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621398    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-099d9ba1e6bd8aabf"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621402    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621404    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621407    2560 flags.go:64] FLAG: --register-node="true"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621410    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621417    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621421    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621424    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621427    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621430    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621433    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 21:45:34.626590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621436    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621439    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621441    2560 flags.go:64] FLAG: --runonce="false"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621444    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621467    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621471    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621474    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621476    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621479    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621482    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621487    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621490    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621493    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621495    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621498    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621501    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621504    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621507    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621512    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621515    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621518    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621522    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621525    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621527    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621530    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 21:45:34.627228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621533    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621536    2560 flags.go:64] FLAG: --v="2"
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621541    2560 flags.go:64] FLAG: --version="false"
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621545    2560 flags.go:64] FLAG: --vmodule=""
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621549    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.621552    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621673    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621676    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621679    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621682    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621685    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621687    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621690    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621693    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621696    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621698    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621701    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621705    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621707    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621710    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621713    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.627873 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621716    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621719    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621722    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621724    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621728    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621732    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621735    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621738    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621741    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621743    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621747    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621749    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621752    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621755    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621757    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621760    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621763    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621766    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621768    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621771    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.628470 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621773    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621776    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621778    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621781    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621783    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621786    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621788    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621791    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621795    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621797    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621800    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621803    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621805    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621808    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621810    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621812    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621815    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621817    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621820    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621823    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.628969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621825    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621827    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621830    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621832    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621835    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621837    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621840    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621842    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621845    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621848    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621851    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621854    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621857    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621859    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621862    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621865    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621867    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621870    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621872    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621875    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.629438 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621878    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621882    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621885    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621888    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621891    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621894    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621897    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621899    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621902    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621905    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621909    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621911    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621914    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621917    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621919    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621922    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621925    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621927    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621930    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.629936 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621932    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621935    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621938    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621941    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.621943    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.622786    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.629835    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.629852    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.629875    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.629881    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629943    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629948    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629952    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629954    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629957    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.630398 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629960    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629962    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629965    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629968    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629970    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629973    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629975    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629978    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629981    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629984    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629987    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629990    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629992    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629995    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.629998    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630001    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630003    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630006    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630008    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.630919 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630011    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630013    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630016    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630018    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630020    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630026    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630030    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630033    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630036    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630038    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630041    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630043    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630046    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630048    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630050    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630053    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630055    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630058    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630061    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.631370 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630063    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630065    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630068    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630071    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630074    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630076    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630079    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630082    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630086    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630089    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630093    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630096    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630099    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630102    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630104    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630107    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630111    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630113    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630116    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.631860 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630119    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630122    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630124    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630127    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630130    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630132    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630135    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630137    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630140    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630142    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630145    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630147    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630150    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630152    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630154    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630157    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630159    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630162    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630164    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630167    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.632327 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630169    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630172    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630174    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630177    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630179    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630181    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630184    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630186    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630189    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630191    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630194    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630197    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630199    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630203    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630205    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630208    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.632869 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630210    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.630215    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630347    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630352    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630355    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630358    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630360    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630363    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630365    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630368    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630371    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630373    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630375    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630378    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630380    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630383    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 21:45:34.633268 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630385    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630387    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630390    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630392    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630396    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630398    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630401    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630403    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630406    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630409    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630412    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630414    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630417    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630420    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630428    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630431    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630434    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630436    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630439    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630441    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 21:45:34.633694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630445    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630463    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630466    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630468    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630471    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630473    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630476    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630478    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630482    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630484    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630487    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630489    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630493    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630496    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630499    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630502    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630505    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630507    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630510    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 21:45:34.634181 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630513    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630515    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630518    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630521    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630524    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630527    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630530    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630532    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630542    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630546    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630555    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630558    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630561    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630564    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630566    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630569    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630572    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630575    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630578    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 21:45:34.634734 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630580    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630583    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630585    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630588    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630590    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630593    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630595    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630598    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630600    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630603    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630607    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630610    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630613    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630615    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630618    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630621    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630624    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630626    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630629    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 21:45:34.635201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630632    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630635    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630637    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630640    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630643    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630645    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630648    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:34.630650    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.630655    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.630797    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.632843    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.634097    2560 server.go:1019] "Starting client certificate rotation"
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.634204    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:45:34.635666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.634240    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 21:45:34.636644 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.636630    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:45:34.638266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.638250    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 21:45:34.649256 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.649237    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 27 21:45:34.652480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.651788    2560 log.go:25] "Validated CRI v1 image API"
Jul 27 21:45:34.653114 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.653093    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 21:45:34.657462 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.657430    2560 fs.go:135] Filesystem UUIDs: map[3d10a344-0b69-4294-bce7-9b36e1d76fef:/dev/nvme0n1p3 4d25905e-2e86-4f67-8622-86ddf0080a10:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 27 21:45:34.657509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.657472    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 21:45:34.663442 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.663309    2560 manager.go:216] Machine: {Timestamp:2026-07-27 21:45:34.661628773 +0000 UTC m=+0.315057405 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102460 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2908b9f06ed2313386a46215d0b41d SystemUUID:ec2908b9-f06e-d231-3386-a46215d0b41d BootID:5f386fe2-4939-4ccc-9ed8-98f7100ffbd8 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:56:c2:8d:57:19 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:56:c2:8d:57:19 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:fd:97:45:bf:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 21:45:34.663442 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.663436    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 21:45:34.663577 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.663536    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 21:45:34.666935 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.666908    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 21:45:34.667104 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.666939    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-110.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 21:45:34.668249 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.668237    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 21:45:34.668293 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.668254    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 21:45:34.668293 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.668262    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:45:34.668940 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.668930    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 21:45:34.670057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.670047    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 21:45:34.670306 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.670290    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:45:34.672619 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672608    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:45:34.672729 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672720    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 21:45:34.672765 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672751    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 21:45:34.672792 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672768    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 21:45:34.672792 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672789    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 21:45:34.672847 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672797    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 27 21:45:34.672847 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.672808    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 21:45:34.673178 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.673162    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:45:34.673178 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.673181    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 21:45:34.674949 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.674933    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 21:45:34.676516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.676500    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 21:45:34.676593 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.676539    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 21:45:34.677841 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677827    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677848    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677858    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677866    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677877    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677885    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677897    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677904    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 21:45:34.677918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677914    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 21:45:34.678154 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677922    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 21:45:34.678154 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677935    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 21:45:34.678154 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.677945    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 21:45:34.678694 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.678683    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 21:45:34.678760 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.678697    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 21:45:34.681927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.681911    2560 server.go:1268] "Started kubelet"
Jul 27 21:45:34.682361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.682007    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 21:45:34.682361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.682104    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 21:45:34.682361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.682142    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 21:45:34.683007 ip-10-0-143-110 systemd[1]: Started Kubernetes Kubelet.
Jul 27 21:45:34.683968 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.683956    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 21:45:34.685705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.685689    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-110.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 21:45:34.686225 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.686196    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 21:45:34.686724 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.686689    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 21:45:34.687151 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.687129    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 21:45:34.688639 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.688619    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 21:45:34.689184 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.688483    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-110.ec2.internal.18c643b223be2118  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-110.ec2.internal,UID:ip-10-0-143-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-110.ec2.internal,},FirstTimestamp:2026-07-27 21:45:34.681882904 +0000 UTC m=+0.335311537,LastTimestamp:2026-07-27 21:45:34.681882904 +0000 UTC m=+0.335311537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-110.ec2.internal,}"
Jul 27 21:45:34.691479 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.691464    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfjxw"
Jul 27 21:45:34.693045 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693033    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 21:45:34.693093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693040    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 21:45:34.693156 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.693129    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:34.693763 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693735    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 21:45:34.693763 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693758    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 21:45:34.693900 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693840    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 21:45:34.693900 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693844    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 21:45:34.693900 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693854    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 21:45:34.694116 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693968    2560 factory.go:55] Registering systemd factory
Jul 27 21:45:34.694116 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.693983    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 27 21:45:34.694589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.694572    2560 factory.go:153] Registering CRI-O factory
Jul 27 21:45:34.694589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.694591    2560 factory.go:223] Registration of the crio container factory successfully
Jul 27 21:45:34.694718 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.694640    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 21:45:34.694718 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.694663    2560 factory.go:103] Registering Raw factory
Jul 27 21:45:34.694718 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.694677    2560 manager.go:1196] Started watching for new ooms in manager
Jul 27 21:45:34.695287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.695271    2560 manager.go:349] Starting recovery of all containers
Jul 27 21:45:34.698169 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.698134    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-110.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 21:45:34.698269 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.698236    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 21:45:34.700411 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.700189    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qfjxw"
Jul 27 21:45:34.703682 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.703649    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 21:45:34.707887 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.707774    2560 manager.go:354] Recovery completed
Jul 27 21:45:34.712022 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.712007    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:34.715338 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715322    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:34.715409 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715355    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:34.715409 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715367    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:34.715934 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715921    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 21:45:34.715934 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715933    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 21:45:34.716015 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715948    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 21:45:34.716015 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.715927    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:34.716736 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.716722    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:34.716806 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.716747    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:34.716806 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.716757    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:34.717874 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.717800    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-110.ec2.internal.18c643b225bc9ef9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-110.ec2.internal,UID:ip-10-0-143-110.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-110.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-110.ec2.internal,},FirstTimestamp:2026-07-27 21:45:34.715338489 +0000 UTC m=+0.368767125,LastTimestamp:2026-07-27 21:45:34.715338489 +0000 UTC m=+0.368767125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-110.ec2.internal,}"
Jul 27 21:45:34.718359 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.718348    2560 policy_none.go:49] "None policy: Start"
Jul 27 21:45:34.718392 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.718365    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 21:45:34.718392 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.718375    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 21:45:34.720475 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.720464    2560 policy_none.go:47] "Start"
Jul 27 21:45:34.757068 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757046    2560 manager.go:337] "Starting Device Plugin manager"
Jul 27 21:45:34.757201 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.757091    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 21:45:34.757201 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757106    2560 server.go:87] "Starting device plugin registration server"
Jul 27 21:45:34.757586 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757573    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 21:45:34.757627 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757588    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 21:45:34.757718 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757695    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 21:45:34.757828 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757777    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 21:45:34.757828 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.757786    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 21:45:34.758332 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.758313    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 21:45:34.758428 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.758362    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:34.858775 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.858694    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:34.860852 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.859693    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:34.860852 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.859731    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:34.860852 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.859743    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:34.860852 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.859774    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-110.ec2.internal"
Jul 27 21:45:34.867701 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.867681    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-110.ec2.internal"
Jul 27 21:45:34.867828 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.867709    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-110.ec2.internal\": node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:34.883401 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.883375    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:34.908979 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.908956    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 21:45:34.909062 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.908997    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 21:45:34.909062 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.909018    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 21:45:34.909062 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.909049    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 21:45:34.910785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:34.910764    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:34.983880 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:34.983850    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.009289 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.009247    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"]
Jul 27 21:45:35.009349 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.009328    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:35.011889 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.011864    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:35.012008 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.011899    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:35.012008 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.011912    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:35.014151 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014138    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:35.014255 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014241    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.014297 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014271    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:35.014949 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014934    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:35.015036 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014956    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:35.015036 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.014965    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:35.015639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.015618    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:35.015723 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.015651    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:35.015723 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.015667    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:35.017865 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.017851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.017928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.017876    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 21:45:35.018533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.018515    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 21:45:35.018615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.018555    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 21:45:35.018615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.018569    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeHasSufficientPID"
Jul 27 21:45:35.045305 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.045273    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-110.ec2.internal\" not found" node="ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.049631 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.049609    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-110.ec2.internal\" not found" node="ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.084178 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.084149    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.095272 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.095245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.095386 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.095280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.095386 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.095304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11d76ec706ed0fd58660b79ceb4322ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-110.ec2.internal\" (UID: \"11d76ec706ed0fd58660b79ceb4322ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.184622 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.184522    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.195925 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.195894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11d76ec706ed0fd58660b79ceb4322ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-110.ec2.internal\" (UID: \"11d76ec706ed0fd58660b79ceb4322ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.195991 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.195933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.195991 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.195959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.196061 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.196001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/11d76ec706ed0fd58660b79ceb4322ce-config\") pod \"kube-apiserver-proxy-ip-10-0-143-110.ec2.internal\" (UID: \"11d76ec706ed0fd58660b79ceb4322ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.196061 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.196022    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.196061 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.196028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5fde131db3559e1d0e86810b1a52b3b9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal\" (UID: \"5fde131db3559e1d0e86810b1a52b3b9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.285092 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.285061    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.347549 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.347524    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.353504 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.353486    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:35.385538 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.385503    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.486186 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.486096    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.586665 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.586629    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.634209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.634182    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 21:45:35.634682 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.634348    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 21:45:35.687595 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.687560    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.706197 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.706157    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 21:40:35 +0000 UTC" deadline="2028-03-27 00:37:44.827944337 +0000 UTC"
Jul 27 21:45:35.706197 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:35.706191    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14594h52m9.12175621s"
Jul 27 21:45:35.787987 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.787901    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.888613 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.888575    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:35.989817 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:35.989778    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-110.ec2.internal\" not found"
Jul 27 21:45:36.012309 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:36.012277    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fde131db3559e1d0e86810b1a52b3b9.slice/crio-416811ada9ec9d0013369362baf410413bddf1b001f9de22a80c80d74db9e56c WatchSource:0}: Error finding container 416811ada9ec9d0013369362baf410413bddf1b001f9de22a80c80d74db9e56c: Status 404 returned error can't find the container with id 416811ada9ec9d0013369362baf410413bddf1b001f9de22a80c80d74db9e56c
Jul 27 21:45:36.016960 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.016943    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:45:36.022650 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:36.022626    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d76ec706ed0fd58660b79ceb4322ce.slice/crio-e7c0fa57c89cf7cb62e0865cff386748dca895c5f100e60295f5ca34735bfc58 WatchSource:0}: Error finding container e7c0fa57c89cf7cb62e0865cff386748dca895c5f100e60295f5ca34735bfc58: Status 404 returned error can't find the container with id e7c0fa57c89cf7cb62e0865cff386748dca895c5f100e60295f5ca34735bfc58
Jul 27 21:45:36.044347 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.044282    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:36.081772 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.081750    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:36.093595 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.093567    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:36.105291 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.105270    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:45:36.106112 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.106098    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal"
Jul 27 21:45:36.114850 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.114831    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 21:45:36.178383 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.178351    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:36.574988 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.574947    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 21:45:36.674110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.673830    2560 apiserver.go:52] "Watching apiserver"
Jul 27 21:45:36.681349 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.681129    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 21:45:36.681655 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.681626    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-ts9l5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal","openshift-multus/multus-ptct7","openshift-multus/network-metrics-daemon-qbjt6","kube-system/konnectivity-agent-dmkp2","kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal","openshift-multus/multus-additional-cni-plugins-ttjzs","openshift-network-diagnostics/network-check-target-qlsm7","openshift-network-operator/iptables-alerter-cfjls","openshift-ovn-kubernetes/ovnkube-node-bmw8w","kube-system/global-pull-secret-syncer-7rm99","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g","openshift-cluster-node-tuning-operator/tuned-wj227","openshift-dns/node-resolver-lfrh2"]
Jul 27 21:45:36.685339 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.684461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.686894 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.686872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.687560 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.687539    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.687686 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.687591    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 21:45:36.688450 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.688421    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 21:45:36.688597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.688468    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 21:45:36.688969 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.688774    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lt4gx\""
Jul 27 21:45:36.688969 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.688888    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.689438 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.689387    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.695556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.691499    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.695556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.694553    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 21:45:36.695556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.694812    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qjf94\""
Jul 27 21:45:36.695556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.695093    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.695556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.695417    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4p4hl\""
Jul 27 21:45:36.695864 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.695753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.695864 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.695838    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:36.696129 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.696109    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 21:45:36.697590 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.696649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.697694 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.697645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.700079 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.699748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.700079 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.699785    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 21:45:36.700471 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.700451    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.700561 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.700519    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jf6d4\""
Jul 27 21:45:36.700961 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.700945    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.701716 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.701695    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 21:45:36.702110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.702091    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.702513 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.702493    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.703051 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703023    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 21:45:36.703236 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-g4ffb\""
Jul 27 21:45:36.703567 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-hostroot\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.703659 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-conf-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.703659 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5756508-46af-4918-82a6-c68756753094-konnectivity-ca\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.703773 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-kubernetes\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.703773 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-run\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.703773 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-cnibin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.703773 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-k8s-cni-cncf-io\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5756508-46af-4918-82a6-c68756753094-agent-certs\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703815    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28v7\" (UniqueName: \"kubernetes.io/projected/800e44b7-5b8c-4c51-af1d-31631bcc2306-kube-api-access-g28v7\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysconfig\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-bin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghn5p\" (UniqueName: \"kubernetes.io/projected/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-kube-api-access-ghn5p\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.703975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.703989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-conf\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-var-lib-kubelet\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704055    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-netns\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704092    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-multus-certs\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-hosts-file\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704159    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7kps\" (UniqueName: \"kubernetes.io/projected/a02c9c60-83e8-4683-a12d-72cca264e74c-kube-api-access-h7kps\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-modprobe-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704285    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-systemd\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-multus-daemon-config\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704350    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btfx\" (UniqueName: \"kubernetes.io/projected/f6658146-9471-4392-9fd7-081a5b3964ae-kube-api-access-5btfx\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a02c9c60-83e8-4683-a12d-72cca264e74c-iptables-alerter-script\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704449    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-os-release\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704553    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-tmp-dir\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-etc-tuned\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdjdl\" (UniqueName: \"kubernetes.io/projected/69929363-6c14-4450-b91a-361632c7b926-kube-api-access-bdjdl\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-system-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-multus\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a02c9c60-83e8-4683-a12d-72cca264e74c-host-slash\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cnibin\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.704811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704805    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wbj\" (UniqueName: \"kubernetes.io/projected/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-kube-api-access-j8wbj\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704826    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-lib-modules\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704842    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-tmp\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-kubelet\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704883    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-etc-kubernetes\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-os-release\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-sys\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-host\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-cni-binary-copy\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.705607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.704974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-socket-dir-parent\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.707237 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.707164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6bcsm\""
Jul 27 21:45:36.707434 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.707415    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.709410 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.709376    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:36.709558 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.709450    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:36.709934 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.709906    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.710041 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.710024    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 21:45:36.710259 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.710244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fccn4\""
Jul 27 21:45:36.710259 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.710254    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.712851 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.712717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.713612 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.713586    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 21:45:36.715542 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.715520    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.715639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.715613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n79rg\""
Jul 27 21:45:36.715735 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.715715    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 21:45:36.715735 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.715730    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 21:45:36.715856 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.715832    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 21:45:36.716176 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.716157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.716257 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.716236    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:36.716439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.716419    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.716689 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.716668    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 21:45:36.719440 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.719417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.722954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.721863    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 21:45:36.722954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.722144    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 21:45:36.722954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.722359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7lhpd\""
Jul 27 21:45:36.722954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.722581    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 21:45:36.795881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.795852    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 21:45:36.805213 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805184    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-log-socket\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.805373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805228    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-env-overrides\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.805373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysconfig\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.805373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-bin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.805373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805319    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.805373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805353    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28v7\" (UniqueName: \"kubernetes.io/projected/800e44b7-5b8c-4c51-af1d-31631bcc2306-kube-api-access-g28v7\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805380    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-conf\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-netns\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a02c9c60-83e8-4683-a12d-72cca264e74c-iptables-alerter-script\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ec5f21-87aa-453b-8b4b-999b7484f0b1-serviceca\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805562    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-netns\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-ovn\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.805670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805668    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-tmp-dir\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ls79\" (UniqueName: \"kubernetes.io/projected/70ec5f21-87aa-453b-8b4b-999b7484f0b1-kube-api-access-8ls79\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-systemd-units\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-slash\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805764    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-systemd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-etc-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805835    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a02c9c60-83e8-4683-a12d-72cca264e74c-host-slash\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cnibin\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-registration-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805970    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-lib-modules\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.805996    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-tmp\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-kubelet\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-etc-kubernetes\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-etc-kubernetes\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806216    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysconfig\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.806278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-bin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.806389    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-kubelet\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-cni-binary-copy\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-socket-dir-parent\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-hostroot\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-conf-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-conf\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-var-lib-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-dbus\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806964    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-cnibin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-k8s-cni-cncf-io\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5756508-46af-4918-82a6-c68756753094-agent-certs\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf74\" (UniqueName: \"kubernetes.io/projected/2834049a-2d33-41d9-95ad-82462e038c11-kube-api-access-drf74\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807075    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghn5p\" (UniqueName: \"kubernetes.io/projected/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-kube-api-access-ghn5p\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.807135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-script-lib\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807153    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-sys-fs\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807185    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-var-lib-kubelet\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-multus-certs\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-hosts-file\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7kps\" (UniqueName: \"kubernetes.io/projected/a02c9c60-83e8-4683-a12d-72cca264e74c-kube-api-access-h7kps\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wbj\" (UniqueName: \"kubernetes.io/projected/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-kube-api-access-j8wbj\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807329    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-node-log\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-netd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-modprobe-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-systemd\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807437    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-multus-daemon-config\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807462    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btfx\" (UniqueName: \"kubernetes.io/projected/f6658146-9471-4392-9fd7-081a5b3964ae-kube-api-access-5btfx\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807484    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807508    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-config\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.807741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-os-release\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807618    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-binary-copy\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807686    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-socket-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-etc-tuned\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807754    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdjdl\" (UniqueName: \"kubernetes.io/projected/69929363-6c14-4450-b91a-361632c7b926-kube-api-access-bdjdl\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807779    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-system-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-sysctl-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.807807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-multus\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.807881    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:37.307846006 +0000 UTC m=+2.961274634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.806935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-netns\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.808343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808328    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a02c9c60-83e8-4683-a12d-72cca264e74c-iptables-alerter-script\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.808863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808446    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.808863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-tmp-dir\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.808863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a02c9c60-83e8-4683-a12d-72cca264e74c-host-slash\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.808958 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cnibin\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.808990    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-systemd\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-var-lib-kubelet\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-conf-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-modprobe-d\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-hosts-file\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809242    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-lib-modules\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-cni-multus\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809403    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-var-lib-kubelet\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-os-release\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809521    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809559    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-cnibin\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-k8s-cni-cncf-io\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-multus-daemon-config\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-system-cni-dir\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-multus-socket-dir-parent\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.810580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.809860    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-host-run-multus-certs\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810082    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-system-cni-dir\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810374    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f6658146-9471-4392-9fd7-081a5b3964ae-cni-binary-copy\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810514    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-bin\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810587    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f6658146-9471-4392-9fd7-081a5b3964ae-hostroot\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-os-release\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2cl\" (UniqueName: \"kubernetes.io/projected/ded94f2e-cde4-4626-82af-d19a92a31f4d-kube-api-access-ps2cl\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-kubelet-config\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-device-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810781    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-sys\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-host\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5756508-46af-4918-82a6-c68756753094-konnectivity-ca\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ec5f21-87aa-453b-8b4b-999b7484f0b1-host\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810958    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.811615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.810985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-kubernetes\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-run\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-run\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-sys\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811268    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-os-release\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-host\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.811609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69929363-6c14-4450-b91a-361632c7b926-etc-kubernetes\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.812470 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.812207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c5756508-46af-4918-82a6-c68756753094-konnectivity-ca\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.813161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.813110    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.813811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.813577    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-tmp\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.813811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.813755    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c5756508-46af-4918-82a6-c68756753094-agent-certs\") pod \"konnectivity-agent-dmkp2\" (UID: \"c5756508-46af-4918-82a6-c68756753094\") " pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:36.814082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.814060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69929363-6c14-4450-b91a-361632c7b926-etc-tuned\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.815716 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.815685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28v7\" (UniqueName: \"kubernetes.io/projected/800e44b7-5b8c-4c51-af1d-31631bcc2306-kube-api-access-g28v7\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:36.821896 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.821866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wbj\" (UniqueName: \"kubernetes.io/projected/7b32e89a-cb35-45f5-b20a-a88c1ab280f2-kube-api-access-j8wbj\") pod \"multus-additional-cni-plugins-ttjzs\" (UID: \"7b32e89a-cb35-45f5-b20a-a88c1ab280f2\") " pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:36.826255 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.823316    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7kps\" (UniqueName: \"kubernetes.io/projected/a02c9c60-83e8-4683-a12d-72cca264e74c-kube-api-access-h7kps\") pod \"iptables-alerter-cfjls\" (UID: \"a02c9c60-83e8-4683-a12d-72cca264e74c\") " pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:36.826255 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.825515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghn5p\" (UniqueName: \"kubernetes.io/projected/6ac976c8-57d2-4848-8d3b-1b331e5eb3b2-kube-api-access-ghn5p\") pod \"node-resolver-lfrh2\" (UID: \"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2\") " pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:36.827140 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.827113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdjdl\" (UniqueName: \"kubernetes.io/projected/69929363-6c14-4450-b91a-361632c7b926-kube-api-access-bdjdl\") pod \"tuned-wj227\" (UID: \"69929363-6c14-4450-b91a-361632c7b926\") " pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:36.828922 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.828893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btfx\" (UniqueName: \"kubernetes.io/projected/f6658146-9471-4392-9fd7-081a5b3964ae-kube-api-access-5btfx\") pod \"multus-ptct7\" (UID: \"f6658146-9471-4392-9fd7-081a5b3964ae\") " pod="openshift-multus/multus-ptct7"
Jul 27 21:45:36.847817 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.847780    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xnzt6"
Jul 27 21:45:36.853396 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.853365    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xnzt6"
Jul 27 21:45:36.911437 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.911437 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-log-socket\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-env-overrides\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ec5f21-87aa-453b-8b4b-999b7484f0b1-serviceca\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911572    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-netns\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-etc-selinux\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-ovn\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-log-socket\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ls79\" (UniqueName: \"kubernetes.io/projected/70ec5f21-87aa-453b-8b4b-999b7484f0b1-kube-api-access-8ls79\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-systemd-units\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.911710 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911704    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-slash\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-systemd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911756    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-etc-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911784    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-registration-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911837    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-kubelet\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-var-lib-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911887    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-dbus\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf74\" (UniqueName: \"kubernetes.io/projected/2834049a-2d33-41d9-95ad-82462e038c11-kube-api-access-drf74\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70ec5f21-87aa-453b-8b4b-999b7484f0b1-serviceca\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911965    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-script-lib\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-netns\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-sys-fs\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-registration-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-etc-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-systemd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.911947    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-systemd-units\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.912209 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-var-lib-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-node-log\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-kubelet\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912114    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-netd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-ovn\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912146    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-slash\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-netd\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-dbus\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-node-log\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-config\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-sys-fs\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.912347    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912336    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-socket-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.912403    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:45:37.412384186 +0000 UTC m=+3.065812811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912469    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.913161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-bin\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-socket-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2cl\" (UniqueName: \"kubernetes.io/projected/ded94f2e-cde4-4626-82af-d19a92a31f4d-kube-api-access-ps2cl\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912567    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-cni-bin\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912570    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-run-openvswitch\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-kubelet-config\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912618    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-device-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ded94f2e-cde4-4626-82af-d19a92a31f4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-env-overrides\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912644    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ec5f21-87aa-453b-8b4b-999b7484f0b1-host\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-device-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70ec5f21-87aa-453b-8b4b-999b7484f0b1-host\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6fe41b7e-2a93-4659-923b-abd00fdabeec-kubelet-config\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.912816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2834049a-2d33-41d9-95ad-82462e038c11-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.914081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.913054    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-script-lib\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.915122 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.913456    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovnkube-config\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.915122 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.915052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal" event={"ID":"11d76ec706ed0fd58660b79ceb4322ce","Type":"ContainerStarted","Data":"e7c0fa57c89cf7cb62e0865cff386748dca895c5f100e60295f5ca34735bfc58"}
Jul 27 21:45:36.916633 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.916607    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ded94f2e-cde4-4626-82af-d19a92a31f4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.916782 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.916640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal" event={"ID":"5fde131db3559e1d0e86810b1a52b3b9","Type":"ContainerStarted","Data":"416811ada9ec9d0013369362baf410413bddf1b001f9de22a80c80d74db9e56c"}
Jul 27 21:45:36.918447 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.918424    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:36.918573 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.918454    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:36.918573 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.918498    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:36.918573 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:36.918558    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:37.418540851 +0000 UTC m=+3.071969498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:36.921311 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.921290    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf74\" (UniqueName: \"kubernetes.io/projected/2834049a-2d33-41d9-95ad-82462e038c11-kube-api-access-drf74\") pod \"aws-ebs-csi-driver-node-rl82g\" (UID: \"2834049a-2d33-41d9-95ad-82462e038c11\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:36.921548 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.921505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2cl\" (UniqueName: \"kubernetes.io/projected/ded94f2e-cde4-4626-82af-d19a92a31f4d-kube-api-access-ps2cl\") pod \"ovnkube-node-bmw8w\" (UID: \"ded94f2e-cde4-4626-82af-d19a92a31f4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:36.921649 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:36.921632    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ls79\" (UniqueName: \"kubernetes.io/projected/70ec5f21-87aa-453b-8b4b-999b7484f0b1-kube-api-access-8ls79\") pod \"node-ca-ts9l5\" (UID: \"70ec5f21-87aa-453b-8b4b-999b7484f0b1\") " pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:37.005827 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.005785    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ttjzs"
Jul 27 21:45:37.018258 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.018219    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wj227"
Jul 27 21:45:37.029469 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.029431    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ptct7"
Jul 27 21:45:37.038606 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.038579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:45:37.045641 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.045612    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lfrh2"
Jul 27 21:45:37.054761 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.054680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cfjls"
Jul 27 21:45:37.064636 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.064610    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ts9l5"
Jul 27 21:45:37.072776 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.072750    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:45:37.079893 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.079843    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g"
Jul 27 21:45:37.315892 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.315858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:37.316093 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.316019    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:37.316093 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.316086    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:38.316071146 +0000 UTC m=+3.969499767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:37.417244 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.417160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:37.417412 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.417303    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:37.417412 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.417368    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:45:38.417353907 +0000 UTC m=+4.070782529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:37.517842 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.517800    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:37.518033 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.517951    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:37.518033 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.517974    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:37.518033 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.517988    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:37.518186 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.518053    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:38.518032045 +0000 UTC m=+4.171460668 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:37.857020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.856978    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:37 +0000 UTC" deadline="2028-02-19 18:37:08.917469331 +0000 UTC"
Jul 27 21:45:37.857020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.857013    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13724h51m31.06046036s"
Jul 27 21:45:37.909657 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.909626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:37.909793 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:37.909626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:37.909793 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.909758    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:37.909875 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:37.909819    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:38.073728 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:38.073666    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5756508_46af_4918_82a6_c68756753094.slice/crio-6d4e266e48033fec8fca807084852aacc031bc3f786d8dd32a5524211623c62a WatchSource:0}: Error finding container 6d4e266e48033fec8fca807084852aacc031bc3f786d8dd32a5524211623c62a: Status 404 returned error can't find the container with id 6d4e266e48033fec8fca807084852aacc031bc3f786d8dd32a5524211623c62a
Jul 27 21:45:38.092694 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:38.092667    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded94f2e_cde4_4626_82af_d19a92a31f4d.slice/crio-93f51bf15ad58f2132055ec46dc0628e99528f0092541b4bdf563c758ecb0dd5 WatchSource:0}: Error finding container 93f51bf15ad58f2132055ec46dc0628e99528f0092541b4bdf563c758ecb0dd5: Status 404 returned error can't find the container with id 93f51bf15ad58f2132055ec46dc0628e99528f0092541b4bdf563c758ecb0dd5
Jul 27 21:45:38.099443 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:38.099417    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ec5f21_87aa_453b_8b4b_999b7484f0b1.slice/crio-721618847a81e0adda40d701e1faa42383ad3f6c47cd786436d607e277289901 WatchSource:0}: Error finding container 721618847a81e0adda40d701e1faa42383ad3f6c47cd786436d607e277289901: Status 404 returned error can't find the container with id 721618847a81e0adda40d701e1faa42383ad3f6c47cd786436d607e277289901
Jul 27 21:45:38.100636 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:38.100590    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6658146_9471_4392_9fd7_081a5b3964ae.slice/crio-9170dc505b41327f9b2af437a7aea045d2ab884e6431d71b58f0ea42c0c0f269 WatchSource:0}: Error finding container 9170dc505b41327f9b2af437a7aea045d2ab884e6431d71b58f0ea42c0c0f269: Status 404 returned error can't find the container with id 9170dc505b41327f9b2af437a7aea045d2ab884e6431d71b58f0ea42c0c0f269
Jul 27 21:45:38.101862 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:45:38.101836    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2834049a_2d33_41d9_95ad_82462e038c11.slice/crio-25e4f79be124c6ed06851219e4c893a32cabcdb2feb6fc084316403ae8cf2c22 WatchSource:0}: Error finding container 25e4f79be124c6ed06851219e4c893a32cabcdb2feb6fc084316403ae8cf2c22: Status 404 returned error can't find the container with id 25e4f79be124c6ed06851219e4c893a32cabcdb2feb6fc084316403ae8cf2c22
Jul 27 21:45:38.325291 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.325095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:38.325437 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.325253    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:38.325437 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.325382    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.325361847 +0000 UTC m=+5.978790469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:38.426100 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.426059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:38.426261 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.426242    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:38.426330 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.426319    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.426302754 +0000 UTC m=+6.079731378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:38.526635 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.526591    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:38.526830 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.526744    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:38.526830 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.526765    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:38.526830 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.526777    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:38.527000 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.526867    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:40.52685087 +0000 UTC m=+6.180279514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:38.858241 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.858199    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 21:40:37 +0000 UTC" deadline="2028-01-08 20:09:21.663236645 +0000 UTC"
Jul 27 21:45:38.858241 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.858236    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12718h23m42.805005162s"
Jul 27 21:45:38.912631 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.910527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:38.912817 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:38.912699    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:38.943206 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.943161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfjls" event={"ID":"a02c9c60-83e8-4683-a12d-72cca264e74c","Type":"ContainerStarted","Data":"252a93f927e8bbb02f71d9af2a004234c9f5a719fd3d1bbc8a1a3685320af8d2"}
Jul 27 21:45:38.948614 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.948577    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerStarted","Data":"de127ca20c9372106d89cf2a371e165009b490e4b3b7f53f3173f259f4b726fe"}
Jul 27 21:45:38.952210 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.952157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ts9l5" event={"ID":"70ec5f21-87aa-453b-8b4b-999b7484f0b1","Type":"ContainerStarted","Data":"721618847a81e0adda40d701e1faa42383ad3f6c47cd786436d607e277289901"}
Jul 27 21:45:38.962559 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.962519    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ptct7" event={"ID":"f6658146-9471-4392-9fd7-081a5b3964ae","Type":"ContainerStarted","Data":"9170dc505b41327f9b2af437a7aea045d2ab884e6431d71b58f0ea42c0c0f269"}
Jul 27 21:45:38.975741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.975701    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lfrh2" event={"ID":"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2","Type":"ContainerStarted","Data":"9024546c1f0c2288028c72e5a45d98d8a5fb8960c6f781641f3104ea2e31d0f9"}
Jul 27 21:45:38.982802 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.982769    2560 generic.go:356] "Generic (PLEG): container finished" podID="5fde131db3559e1d0e86810b1a52b3b9" containerID="f2d0858aabca7247895d1f458c91c7d8ae485b33c72ddd213364b4d7f960cda7" exitCode=0
Jul 27 21:45:38.983104 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.983077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal" event={"ID":"5fde131db3559e1d0e86810b1a52b3b9","Type":"ContainerDied","Data":"f2d0858aabca7247895d1f458c91c7d8ae485b33c72ddd213364b4d7f960cda7"}
Jul 27 21:45:38.988802 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.988763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wj227" event={"ID":"69929363-6c14-4450-b91a-361632c7b926","Type":"ContainerStarted","Data":"62ae54585d9b628866e38aa95f4ca74d1d78b3f244a311e9a78dce7c0f3d34bc"}
Jul 27 21:45:38.995569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:38.995542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g" event={"ID":"2834049a-2d33-41d9-95ad-82462e038c11","Type":"ContainerStarted","Data":"25e4f79be124c6ed06851219e4c893a32cabcdb2feb6fc084316403ae8cf2c22"}
Jul 27 21:45:39.012170 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:39.012139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"93f51bf15ad58f2132055ec46dc0628e99528f0092541b4bdf563c758ecb0dd5"}
Jul 27 21:45:39.019078 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:39.016792    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmkp2" event={"ID":"c5756508-46af-4918-82a6-c68756753094","Type":"ContainerStarted","Data":"6d4e266e48033fec8fca807084852aacc031bc3f786d8dd32a5524211623c62a"}
Jul 27 21:45:39.034566 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:39.034516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal" event={"ID":"11d76ec706ed0fd58660b79ceb4322ce","Type":"ContainerStarted","Data":"601dd83262afd80bede7736a70c14efa41fd8ba2e92cd197bfe026d291ab5e64"}
Jul 27 21:45:39.909790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:39.909752    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:39.910300 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:39.909899    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:39.910365 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:39.910324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:39.910433 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:39.910416    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:40.059259 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.059222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal" event={"ID":"5fde131db3559e1d0e86810b1a52b3b9","Type":"ContainerStarted","Data":"fdc31274a5b5afbd6aa752dc3d18c8c84ac6714b35bb16c3a9b8b168b873f55c"}
Jul 27 21:45:40.074993 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.074925    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-110.ec2.internal" podStartSLOduration=4.07490372 podStartE2EDuration="4.07490372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:39.050780401 +0000 UTC m=+4.704209047" watchObservedRunningTime="2026-07-27 21:45:40.07490372 +0000 UTC m=+5.728332368"
Jul 27 21:45:40.344558 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.343736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:40.344558 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.343986    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:40.344558 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.344066    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:44.344046284 +0000 UTC m=+9.997474911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:40.445090 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.444388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:40.445090 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.444593    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:40.445090 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.444665    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:45:44.444645297 +0000 UTC m=+10.098073921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:40.546363 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.545649    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:40.546363 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.545860    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:40.546363 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.545881    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:40.546363 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.545895    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:40.546363 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.545959    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:44.545939134 +0000 UTC m=+10.199367758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:40.912755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:40.912721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:40.913237 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:40.912887    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:41.909227 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:41.909192    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:41.909420 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:41.909310    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:41.909711 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:41.909675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:41.909882 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:41.909815    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:42.911300 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:42.911259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:42.911808 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:42.911409    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:43.909821 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:43.909568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:43.909821 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:43.909695    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:43.909821 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:43.909728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:43.909821 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:43.909787    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:44.381187 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:44.381150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:44.381692 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.381286    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:44.381692 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.381349    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.381334599 +0000 UTC m=+18.034763220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:44.482266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:44.482233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:44.482589 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.482407    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:44.482589 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.482499    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.482479294 +0000 UTC m=+18.135907929 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:44.583320 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:44.582711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:44.583320 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.582873    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:44.583320 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.582896    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:44.583320 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.582910    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:44.583320 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.582967    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:45:52.582952095 +0000 UTC m=+18.236380737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:44.911326 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:44.911290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:44.911537 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:44.911442    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:45.909979 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:45.909943    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:45.910439 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:45.910088    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:45.910439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:45.910153    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:45.910439 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:45.910218    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:46.910261 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:46.909740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:46.910261 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:46.909885    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:47.910238 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:47.910206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:47.910427 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:47.910206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:47.910427 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:47.910313    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:47.910427 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:47.910409    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:48.909471 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:48.909417    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:48.909646 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:48.909588    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:49.910245 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:49.910169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:49.910623 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:49.910309    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:49.910623 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:49.910169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:49.910720 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:49.910693    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:50.910353 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:50.910318    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:50.910894 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:50.910445    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:51.909715 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:51.909678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:51.909931 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:51.909806    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:51.910251 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:51.910223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:51.910362 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:51.910316    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:52.442522 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:52.442482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:52.443003 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.442616    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:52.443003 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.442682    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:08.442666705 +0000 UTC m=+34.096095325 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:45:52.543396 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:52.543341    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:52.543582 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.543517    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:52.543636 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.543623    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:46:08.543577212 +0000 UTC m=+34.197005832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:45:52.644472 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:52.644419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:52.644631 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.644602    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:45:52.644631 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.644627    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:45:52.644738 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.644637    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:52.644738 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.644703    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:08.6446888 +0000 UTC m=+34.298117420 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:45:52.909574 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:52.909538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:52.909720 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:52.909665    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:53.909816 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:53.909779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:53.910215 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:53.909901    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:53.910215 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:53.909787    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:53.910318 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:53.910294    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:54.910556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:54.910515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:54.911014 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:54.910653    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:55.909987 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:55.909954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:55.910157 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:55.909964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:55.910157 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:55.910062    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:55.910260 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:55.910164    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:56.909938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:56.909902    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:56.910632 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:56.910106    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:57.909736 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:57.909566    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:57.909829 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:57.909635    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:57.909866 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:57.909817    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:57.909956 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:57.909934    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:45:58.098135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.098098    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="8e7e3a4371281fa70b7aa5fab3220bd5ee08f589a6e419c539d447a80dd7cc2f" exitCode=0
Jul 27 21:45:58.099053 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.098200    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"8e7e3a4371281fa70b7aa5fab3220bd5ee08f589a6e419c539d447a80dd7cc2f"}
Jul 27 21:45:58.100120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.100089    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ts9l5" event={"ID":"70ec5f21-87aa-453b-8b4b-999b7484f0b1","Type":"ContainerStarted","Data":"f7d420708d9c61ce2016fcb2e294b3a0da0aa3b02074551b761bbd4d04304c97"}
Jul 27 21:45:58.102071 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.102044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ptct7" event={"ID":"f6658146-9471-4392-9fd7-081a5b3964ae","Type":"ContainerStarted","Data":"f481a3c0f7d3c9cbcfd0a0a25e90af34a8cb9b19a6429d8ff425c02ca0fa94aa"}
Jul 27 21:45:58.103583 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.103560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lfrh2" event={"ID":"6ac976c8-57d2-4848-8d3b-1b331e5eb3b2","Type":"ContainerStarted","Data":"64476b940aae3691ce205b4a35c75bebfee96aa142bd09b4d536c9afb6aa0196"}
Jul 27 21:45:58.105288 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.105264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wj227" event={"ID":"69929363-6c14-4450-b91a-361632c7b926","Type":"ContainerStarted","Data":"2be62d17df9e1ca2b780fff69596bbab39279eaeb6791afa991d377ce3ccef20"}
Jul 27 21:45:58.106844 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.106817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g" event={"ID":"2834049a-2d33-41d9-95ad-82462e038c11","Type":"ContainerStarted","Data":"fae9820bb7116e8396e2e7509781b68975c7307962b8146f1f1a7ec76eb1b47c"}
Jul 27 21:45:58.110364 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"9df591a56f2675d683545a389ad04c465aea879f9e5968fdc0115a949fc3e4a4"}
Jul 27 21:45:58.110499 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"e4db24d05b815cebba04ed723d0de0cab4314c247f4c5acc5fb607cbcc663843"}
Jul 27 21:45:58.110568 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"e58a952ebb395712c317bb165e5d410004f8810690a492580614109b94268f2c"}
Jul 27 21:45:58.110568 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110547    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"16b5ae82a62601ba4644769294a1d2c493649e4310063068181cc52e222e065b"}
Jul 27 21:45:58.110676 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"4d42c2196d0e3add472beb6ea3bc68cb24760c974dea9b07ff00ec1fc237f7f4"}
Jul 27 21:45:58.110676 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.110643    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"e890a27e1a978e39df2b8ab03f5689037293bce38ec114d20253fc214aa9786e"}
Jul 27 21:45:58.111853 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.111826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dmkp2" event={"ID":"c5756508-46af-4918-82a6-c68756753094","Type":"ContainerStarted","Data":"2494a9ea20500fcb202efaa1d7056aae789781c17b079aead0a07460c82e4495"}
Jul 27 21:45:58.126389 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.126328    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-110.ec2.internal" podStartSLOduration=22.126307004 podStartE2EDuration="22.126307004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:40.07584798 +0000 UTC m=+5.729276627" watchObservedRunningTime="2026-07-27 21:45:58.126307004 +0000 UTC m=+23.779735639"
Jul 27 21:45:58.139785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.139682    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dmkp2" podStartSLOduration=3.839151386 podStartE2EDuration="23.139665067s" totalImagesPullingTime="19.300513681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:58.139359308 +0000 UTC m=+23.792787950" watchObservedRunningTime="2026-07-27 21:45:58.139665067 +0000 UTC m=+23.793093708"
Jul 27 21:45:58.153190 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.153133    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ts9l5" podStartSLOduration=3.865872591 podStartE2EDuration="23.153115131s" totalImagesPullingTime="19.28724254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:58.152875484 +0000 UTC m=+23.806304125" watchObservedRunningTime="2026-07-27 21:45:58.153115131 +0000 UTC m=+23.806543772"
Jul 27 21:45:58.186508 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.186437    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wj227" podStartSLOduration=3.903594094 podStartE2EDuration="23.186422132s" totalImagesPullingTime="19.282828038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:58.185961614 +0000 UTC m=+23.839390255" watchObservedRunningTime="2026-07-27 21:45:58.186422132 +0000 UTC m=+23.839850774"
Jul 27 21:45:58.186667 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.186555    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lfrh2" podStartSLOduration=3.880804144 podStartE2EDuration="23.186548894s" totalImagesPullingTime="19.30574475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:58.167173373 +0000 UTC m=+23.820602011" watchObservedRunningTime="2026-07-27 21:45:58.186548894 +0000 UTC m=+23.839977537"
Jul 27 21:45:58.202343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.202299    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ptct7" podStartSLOduration=3.851122885 podStartE2EDuration="23.202282441s" totalImagesPullingTime="19.351159556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:58.202209495 +0000 UTC m=+23.855638139" watchObservedRunningTime="2026-07-27 21:45:58.202282441 +0000 UTC m=+23.855711079"
Jul 27 21:45:58.860354 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.860289    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 21:45:58.909810 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:58.909777    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:45:58.910042 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:58.909909    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:45:59.116378 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.116335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g" event={"ID":"2834049a-2d33-41d9-95ad-82462e038c11","Type":"ContainerStarted","Data":"7a86413957abd9d26fe918c0b4fdb02dc5d5db1b6f3a34ec68876d24d7edae5f"}
Jul 27 21:45:59.117922 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.117886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cfjls" event={"ID":"a02c9c60-83e8-4683-a12d-72cca264e74c","Type":"ContainerStarted","Data":"74dc5bf7970e67b5fd485130ad274db31b8d834502df378841e91101b23afe60"}
Jul 27 21:45:59.132799 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.132730    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cfjls" podStartSLOduration=4.836717919 podStartE2EDuration="24.132710449s" totalImagesPullingTime="19.29599253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:45:59.132530983 +0000 UTC m=+24.785959624" watchObservedRunningTime="2026-07-27 21:45:59.132710449 +0000 UTC m=+24.786139090"
Jul 27 21:45:59.774657 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.774529    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T21:45:58.860313321Z","UUID":"9e4c864c-7188-4698-bf56-b699ecc6b31e","Handler":null,"Name":"","Endpoint":""}
Jul 27 21:45:59.777061 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.777036    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 21:45:59.777232 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.777083    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 21:45:59.909655 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.909626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:45:59.909835 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:45:59.909630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:45:59.909835 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:59.909756    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:45:59.909835 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:45:59.909818    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:00.775440 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:00.775407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:46:00.776048 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:00.776031    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:46:00.910307 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:00.910276    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:00.910507 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:00.910430    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:01.125232 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.125195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g" event={"ID":"2834049a-2d33-41d9-95ad-82462e038c11","Type":"ContainerStarted","Data":"7b091f058175b4e24ab98d9cb9152a0b2d47b5c78a8659b30fb94a5a4f4303a6"}
Jul 27 21:46:01.129040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.128998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"17f91d68697ee2423fb2706dd5095a5d7441463e927a337a0167c96d8f15fecf"}
Jul 27 21:46:01.129194 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.129083    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:46:01.129688 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.129667    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dmkp2"
Jul 27 21:46:01.151123 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.151074    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rl82g" podStartSLOduration=4.579862549 podStartE2EDuration="26.151056998s" totalImagesPullingTime="21.571194449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:01.150779772 +0000 UTC m=+26.804208426" watchObservedRunningTime="2026-07-27 21:46:01.151056998 +0000 UTC m=+26.804485641"
Jul 27 21:46:01.909828 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.909609    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:01.910262 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:01.909611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:01.910262 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:01.909922    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:01.910262 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:01.910001    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:02.909964 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:02.909929    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:02.910732 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:02.910073    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:03.135767 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:03.135731    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="fce9d7362b554e7fb83a85a073513d600532d8ccb1b4d5d29c68968a821f60f0" exitCode=0
Jul 27 21:46:03.135950 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:03.135823    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"fce9d7362b554e7fb83a85a073513d600532d8ccb1b4d5d29c68968a821f60f0"}
Jul 27 21:46:03.139563 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:03.139536    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" event={"ID":"ded94f2e-cde4-4626-82af-d19a92a31f4d","Type":"ContainerStarted","Data":"66715b4862b474302fb634f3d8a18cc8bfe1f817265491a4fedc05a02fa70d1b"}
Jul 27 21:46:03.910235 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:03.910040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:03.910778 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:03.910041    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:03.910778 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:03.910331    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:03.910778 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:03.910417    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:04.142890 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.142859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:04.143057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.142977    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:04.143057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.143021    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:04.161044 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.160945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:04.161399 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.161379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:04.192286 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.192228    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w" podStartSLOduration=9.827401603 podStartE2EDuration="29.192208876s" totalImagesPullingTime="19.364807273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:03.198551613 +0000 UTC m=+28.851980258" watchObservedRunningTime="2026-07-27 21:46:04.192208876 +0000 UTC m=+29.845637519"
Jul 27 21:46:04.514480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.514377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rm99"]
Jul 27 21:46:04.514669 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.514530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:04.514669 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:04.514653    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:04.518354 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.518327    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qlsm7"]
Jul 27 21:46:04.518550 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.518436    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:04.518616 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:04.518583    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:04.518986 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.518965    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbjt6"]
Jul 27 21:46:04.519082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:04.519069    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:04.519190 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:04.519170    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:05.146268 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:05.146235    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="5f0ce6a98b827b2a97febb1b5945d714b6288bc714b5c830fb77ba096422c250" exitCode=0
Jul 27 21:46:05.146763 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:05.146325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"5f0ce6a98b827b2a97febb1b5945d714b6288bc714b5c830fb77ba096422c250"}
Jul 27 21:46:05.910125 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:05.910093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:05.910283 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:05.910200    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:05.910283 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:05.910250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:05.910357 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:05.910339    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:06.909754 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:06.909582    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:06.910128 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:06.909848    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:07.152591 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:07.152553    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="2ffa3e7c07f6d9c1c3db6289b4c1294d2eb032614166b729dc35d52a0147b148" exitCode=0
Jul 27 21:46:07.152781 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:07.152615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"2ffa3e7c07f6d9c1c3db6289b4c1294d2eb032614166b729dc35d52a0147b148"}
Jul 27 21:46:07.909701 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:07.909663    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:07.909701 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:07.909706    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:07.910189 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:07.909798    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:07.910189 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:07.909908    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:08.457720 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:08.457679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:08.457900 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.457841    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:46:08.457967 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.457907    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs podName:800e44b7-5b8c-4c51-af1d-31631bcc2306 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:40.457885599 +0000 UTC m=+66.111314240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs") pod "network-metrics-daemon-qbjt6" (UID: "800e44b7-5b8c-4c51-af1d-31631bcc2306") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 21:46:08.558494 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:08.558438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:08.558690 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.558586    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 21:46:08.558690 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.558664    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret podName:6fe41b7e-2a93-4659-923b-abd00fdabeec nodeName:}" failed. No retries permitted until 2026-07-27 21:46:40.55864493 +0000 UTC m=+66.212073568 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret") pod "global-pull-secret-syncer-7rm99" (UID: "6fe41b7e-2a93-4659-923b-abd00fdabeec") : object "kube-system"/"original-pull-secret" not registered
Jul 27 21:46:08.659380 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:08.659338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:08.659597 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.659559    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 21:46:08.659597 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.659584    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 21:46:08.659597 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.659595    2560 projected.go:196] Error preparing data for projected volume kube-api-access-65bp5 for pod openshift-network-diagnostics/network-check-target-qlsm7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:46:08.659750 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.659657    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5 podName:2ba9ea4a-bac1-44da-b1af-052c8d54f737 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:40.659641969 +0000 UTC m=+66.313070597 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-65bp5" (UniqueName: "kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5") pod "network-check-target-qlsm7" (UID: "2ba9ea4a-bac1-44da-b1af-052c8d54f737") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 21:46:08.910385 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:08.910345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:08.910863 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:08.910512    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:09.909303 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:09.909266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:09.909303 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:09.909316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:09.909568 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:09.909424    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:09.909629 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:09.909565    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:10.909268 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:10.909235    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:10.909834 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:10.909377    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:11.909328 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:11.909294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:11.909831 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:11.909294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:11.909831 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:11.909412    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:11.909831 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:11.909500    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:12.167090 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:12.167001    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="e4e31bdda50d8420f8e213468713f604d17adc18441a2a20ac6c99e09a691f23" exitCode=0
Jul 27 21:46:12.167090 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:12.167070    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"e4e31bdda50d8420f8e213468713f604d17adc18441a2a20ac6c99e09a691f23"}
Jul 27 21:46:12.910289 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:12.910256    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:12.910793 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:12.910374    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:13.172901 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:13.172810    2560 generic.go:356] "Generic (PLEG): container finished" podID="7b32e89a-cb35-45f5-b20a-a88c1ab280f2" containerID="cecf1b7cd960d3057ecb51f07be67e177707e65e89f3def54eff5fe613921315" exitCode=0
Jul 27 21:46:13.172901 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:13.172848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerDied","Data":"cecf1b7cd960d3057ecb51f07be67e177707e65e89f3def54eff5fe613921315"}
Jul 27 21:46:13.909487 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:13.909435    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:13.909660 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:13.909557    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:13.909660 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:13.909608    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:13.909745 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:13.909706    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:14.178547 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:14.178469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" event={"ID":"7b32e89a-cb35-45f5-b20a-a88c1ab280f2","Type":"ContainerStarted","Data":"3c68a37eb4135bed562c52ecd9d4f1d70f7261c91ec0be20de8898f68b281377"}
Jul 27 21:46:14.200623 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:14.200562    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ttjzs" podStartSLOduration=9.197691394 podStartE2EDuration="39.200544367s" totalImagesPullingTime="30.002852973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:14.200273732 +0000 UTC m=+39.853702374" watchObservedRunningTime="2026-07-27 21:46:14.200544367 +0000 UTC m=+39.853973008"
Jul 27 21:46:14.910388 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:14.910350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:14.910582 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:14.910441    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:15.909915 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:15.909876    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:15.909915 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:15.909920    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:15.910348 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:15.909987    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:15.910348 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:15.910044    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:16.909418 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:16.909384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:16.909607 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:16.909551    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:17.910224 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:17.910189    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:17.910609 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:17.910188    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:17.910609 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:17.910301    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:17.910609 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:17.910383    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:18.909473 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:18.909420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:18.909663 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:18.909559    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:19.910114 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:19.910077    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:19.910603 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:19.910081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:19.910603 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:19.910183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:19.910603 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:19.910275    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:20.909551 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:20.909515    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:20.909716 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:20.909627    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:21.909679 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:21.909645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:21.910064 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:21.909648    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:21.910064 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:21.909751    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:21.910064 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:21.909820    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:22.909524 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:22.909494    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:22.909691 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:22.909607    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qbjt6" podUID="800e44b7-5b8c-4c51-af1d-31631bcc2306"
Jul 27 21:46:23.909621 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:23.909589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:23.909990 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:23.909606    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:23.909990 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:23.909722    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7rm99" podUID="6fe41b7e-2a93-4659-923b-abd00fdabeec"
Jul 27 21:46:23.909990 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:23.909754    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qlsm7" podUID="2ba9ea4a-bac1-44da-b1af-052c8d54f737"
Jul 27 21:46:24.212079 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.212003    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-110.ec2.internal" event="NodeReady"
Jul 27 21:46:24.212214 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.212128    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 21:46:24.272605 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.272575    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8m9h8"]
Jul 27 21:46:24.291240 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.291213    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tj4z9"]
Jul 27 21:46:24.291412 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.291344    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.294406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.294385    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 21:46:24.294560 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.294417    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 21:46:24.294640 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.294627    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jb57h\""
Jul 27 21:46:24.315229 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.315205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8m9h8"]
Jul 27 21:46:24.315365 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.315245    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tj4z9"]
Jul 27 21:46:24.315365 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.315331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.319005 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.318977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-b6tfk\""
Jul 27 21:46:24.319130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.319006    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 21:46:24.319130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.319022    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 21:46:24.319130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.319056    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 21:46:24.477213 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-cert\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.477213 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnnzs\" (UniqueName: \"kubernetes.io/projected/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-kube-api-access-dnnzs\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.477395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477243    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4e105-b378-4b64-8540-fb84c7d86ac9-config-volume\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.477395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrqw\" (UniqueName: \"kubernetes.io/projected/52d4e105-b378-4b64-8540-fb84c7d86ac9-kube-api-access-mzrqw\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.477395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52d4e105-b378-4b64-8540-fb84c7d86ac9-metrics-tls\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.477395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.477341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4e105-b378-4b64-8540-fb84c7d86ac9-tmp-dir\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.578516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4e105-b378-4b64-8540-fb84c7d86ac9-config-volume\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.578516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrqw\" (UniqueName: \"kubernetes.io/projected/52d4e105-b378-4b64-8540-fb84c7d86ac9-kube-api-access-mzrqw\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.578705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578646    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52d4e105-b378-4b64-8540-fb84c7d86ac9-metrics-tls\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.578705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4e105-b378-4b64-8540-fb84c7d86ac9-tmp-dir\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.578705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-cert\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.578849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.578808    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnnzs\" (UniqueName: \"kubernetes.io/projected/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-kube-api-access-dnnzs\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.579108 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.579080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4e105-b378-4b64-8540-fb84c7d86ac9-tmp-dir\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.579204 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.579089    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/52d4e105-b378-4b64-8540-fb84c7d86ac9-config-volume\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.582785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.582762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/52d4e105-b378-4b64-8540-fb84c7d86ac9-metrics-tls\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.582882 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.582805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-cert\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.587875 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.587852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnnzs\" (UniqueName: \"kubernetes.io/projected/7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca-kube-api-access-dnnzs\") pod \"ingress-canary-tj4z9\" (UID: \"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca\") " pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.598934 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.598911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrqw\" (UniqueName: \"kubernetes.io/projected/52d4e105-b378-4b64-8540-fb84c7d86ac9-kube-api-access-mzrqw\") pod \"dns-default-8m9h8\" (UID: \"52d4e105-b378-4b64-8540-fb84c7d86ac9\") " pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.600865 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.600850    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:24.624111 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.624083    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tj4z9"
Jul 27 21:46:24.794774 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.794744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8m9h8"]
Jul 27 21:46:24.808415 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:24.808380    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4e105_b378_4b64_8540_fb84c7d86ac9.slice/crio-3c7fcc62405d584cae1b4dea0e0769e20120d302ea9f541ce488ec04ae363799 WatchSource:0}: Error finding container 3c7fcc62405d584cae1b4dea0e0769e20120d302ea9f541ce488ec04ae363799: Status 404 returned error can't find the container with id 3c7fcc62405d584cae1b4dea0e0769e20120d302ea9f541ce488ec04ae363799
Jul 27 21:46:24.809360 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.809337    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tj4z9"]
Jul 27 21:46:24.812406 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:24.812382    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4cbc3c_bf7c_4dbd_94f2_3e9e3caef4ca.slice/crio-71a8b203782b1e2687da46fa4cb4753116d87c47b69e9b955f5cff57b31ea8bc WatchSource:0}: Error finding container 71a8b203782b1e2687da46fa4cb4753116d87c47b69e9b955f5cff57b31ea8bc: Status 404 returned error can't find the container with id 71a8b203782b1e2687da46fa4cb4753116d87c47b69e9b955f5cff57b31ea8bc
Jul 27 21:46:24.912586 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.912557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:24.915468 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.915428    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:46:24.915995 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:24.915897    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\""
Jul 27 21:46:25.204865 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.204817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m9h8" event={"ID":"52d4e105-b378-4b64-8540-fb84c7d86ac9","Type":"ContainerStarted","Data":"3c7fcc62405d584cae1b4dea0e0769e20120d302ea9f541ce488ec04ae363799"}
Jul 27 21:46:25.205926 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.205898    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tj4z9" event={"ID":"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca","Type":"ContainerStarted","Data":"71a8b203782b1e2687da46fa4cb4753116d87c47b69e9b955f5cff57b31ea8bc"}
Jul 27 21:46:25.910350 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.910311    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:25.910350 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.910331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:25.914970 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.914944    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:46:25.915358 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.915223    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\""
Jul 27 21:46:25.915358 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.915228    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:46:25.915683 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:25.915659    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:46:28.215713 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.215617    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tj4z9" event={"ID":"7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca","Type":"ContainerStarted","Data":"cf4850966212e17a5949f4cd487e9ce3581662162682000e7a27e0f5a6847eb0"}
Jul 27 21:46:28.217383 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.217359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m9h8" event={"ID":"52d4e105-b378-4b64-8540-fb84c7d86ac9","Type":"ContainerStarted","Data":"00f73c60bb5a91c451656d63d3e73d27b2bb6a581db579b454b9c9e5dd1c42f2"}
Jul 27 21:46:28.217528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.217398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8m9h8" event={"ID":"52d4e105-b378-4b64-8540-fb84c7d86ac9","Type":"ContainerStarted","Data":"329048971f398c834b2713779aa63f4109449219bb2b81eea53ba8039a86c8a4"}
Jul 27 21:46:28.217528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.217500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:28.233388 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.233334    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tj4z9" podStartSLOduration=1.176212292 podStartE2EDuration="4.233318045s" totalImagesPullingTime="3.057105753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:28.232082774 +0000 UTC m=+53.885511426" watchObservedRunningTime="2026-07-27 21:46:28.233318045 +0000 UTC m=+53.886746686"
Jul 27 21:46:28.256087 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:28.256031    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8m9h8" podStartSLOduration=1.200193398 podStartE2EDuration="4.256014563s" totalImagesPullingTime="3.055821165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:28.255303483 +0000 UTC m=+53.908732127" watchObservedRunningTime="2026-07-27 21:46:28.256014563 +0000 UTC m=+53.909443184"
Jul 27 21:46:29.304039 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.304006    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"]
Jul 27 21:46:29.305701 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.305682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:29.308430 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.308413    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 27 21:46:29.308561 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.308490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-q2cth\""
Jul 27 21:46:29.311124 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.311091    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9mqs\" (UID: \"7f7d3091-25db-4808-ad26-da8555fdccf3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:29.324302 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.324268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"]
Jul 27 21:46:29.412133 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.412093    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9mqs\" (UID: \"7f7d3091-25db-4808-ad26-da8555fdccf3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:29.412345 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.412276    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:46:29.412415 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.412383    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates podName:7f7d3091-25db-4808-ad26-da8555fdccf3 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:29.912361324 +0000 UTC m=+55.565789947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-m9mqs" (UID: "7f7d3091-25db-4808-ad26-da8555fdccf3") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 27 21:46:29.556963 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.556930    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qm5vp"]
Jul 27 21:46:29.558695 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.558679    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.561461 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.561421    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 21:46:29.561461 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.561432    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 21:46:29.561633 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.561513    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 21:46:29.561801 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.561783    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7smr8\""
Jul 27 21:46:29.562004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.561988    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 21:46:29.569440 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.569420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qm5vp"]
Jul 27 21:46:29.613545 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.613511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z62l\" (UniqueName: \"kubernetes.io/projected/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-api-access-7z62l\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.613728 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.613614    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.613728 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.613647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71c0cb96-1bc0-411c-88bf-2137aae6a264-data-volume\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.613728 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.613687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71c0cb96-1bc0-411c-88bf-2137aae6a264-crio-socket\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.613728 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.613706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.713986 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.713947    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.713986 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.713989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71c0cb96-1bc0-411c-88bf-2137aae6a264-data-volume\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.714012    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71c0cb96-1bc0-411c-88bf-2137aae6a264-crio-socket\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.714027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.714065    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.714145    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.214130178 +0000 UTC m=+55.867558798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.714166    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.714074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z62l\" (UniqueName: \"kubernetes.io/projected/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-api-access-7z62l\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:29.714217    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:30.214200889 +0000 UTC m=+55.867629528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.714225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/71c0cb96-1bc0-411c-88bf-2137aae6a264-crio-socket\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.714280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.714247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/71c0cb96-1bc0-411c-88bf-2137aae6a264-data-volume\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.722657 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.722629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z62l\" (UniqueName: \"kubernetes.io/projected/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-api-access-7z62l\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:29.914823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.914731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9mqs\" (UID: \"7f7d3091-25db-4808-ad26-da8555fdccf3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:29.917296 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.917265    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7f7d3091-25db-4808-ad26-da8555fdccf3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-m9mqs\" (UID: \"7f7d3091-25db-4808-ad26-da8555fdccf3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:29.917426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:29.917365    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:30.008572 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.008545    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:46:30.010614 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.010590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.013350 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013317    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 21:46:30.013350 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013326    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 21:46:30.013594 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013575    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bvpmm\""
Jul 27 21:46:30.013707 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 21:46:30.013781 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013768    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 21:46:30.013955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.013943    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 21:46:30.015023 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015004    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 21:46:30.015072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015029    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 21:46:30.015072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.015160 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.015160 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.015160 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgk7b\" (UniqueName: \"kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.015266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.015319 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.015263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.024893 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.024866    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:46:30.065350 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.065263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"]
Jul 27 21:46:30.069661 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:30.069635    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7d3091_25db_4808_ad26_da8555fdccf3.slice/crio-98fc8923f2a378c463a62a0a8702b85f9b7287672137a9807bd19c4b29440386 WatchSource:0}: Error finding container 98fc8923f2a378c463a62a0a8702b85f9b7287672137a9807bd19c4b29440386: Status 404 returned error can't find the container with id 98fc8923f2a378c463a62a0a8702b85f9b7287672137a9807bd19c4b29440386
Jul 27 21:46:30.115648 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.115849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115672    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.115849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.115849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.115849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgk7b\" (UniqueName: \"kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.115849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.115789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.116408 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.116385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.116528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.116423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.116528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.116405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.118274 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.118249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.118364 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.118308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.123997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.123974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgk7b\" (UniqueName: \"kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b\") pod \"console-cf788c7df-hflhf\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") " pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.187010 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.186935    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m9h8_52d4e105-b378-4b64-8540-fb84c7d86ac9/dns/0.log"
Jul 27 21:46:30.216687 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.216655    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:30.216778 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.216702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:30.216778 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:30.216769    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:30.216870 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:30.216816    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:30.216870 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:30.216831    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.216816528 +0000 UTC m=+56.870245148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:30.216870 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:30.216853    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:31.216842406 +0000 UTC m=+56.870271025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:30.222696 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.222672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs" event={"ID":"7f7d3091-25db-4808-ad26-da8555fdccf3","Type":"ContainerStarted","Data":"98fc8923f2a378c463a62a0a8702b85f9b7287672137a9807bd19c4b29440386"}
Jul 27 21:46:30.321845 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.321812    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:30.373799 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.373777    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m9h8_52d4e105-b378-4b64-8540-fb84c7d86ac9/kube-rbac-proxy/0.log"
Jul 27 21:46:30.444741 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:30.444664    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:46:30.449856 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:30.449780    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3df6ff7_95a5_401c_9919_f7b2b7e0fe55.slice/crio-539d083bf483d39e7259e3b9ae9e8cab272f2b2ed90921a4d7b08a09d9980b35 WatchSource:0}: Error finding container 539d083bf483d39e7259e3b9ae9e8cab272f2b2ed90921a4d7b08a09d9980b35: Status 404 returned error can't find the container with id 539d083bf483d39e7259e3b9ae9e8cab272f2b2ed90921a4d7b08a09d9980b35
Jul 27 21:46:31.226279 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:31.226244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:31.226492 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:31.226292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:31.226492 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:31.226367    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:31.226492 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:31.226426    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.226408831 +0000 UTC m=+58.879837468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:31.226492 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:31.226428    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:31.226492 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:31.226485    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:33.226474458 +0000 UTC m=+58.879903078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:31.228321 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:31.228291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf788c7df-hflhf" event={"ID":"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55","Type":"ContainerStarted","Data":"539d083bf483d39e7259e3b9ae9e8cab272f2b2ed90921a4d7b08a09d9980b35"}
Jul 27 21:46:31.770239 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:31.770205    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lfrh2_6ac976c8-57d2-4848-8d3b-1b331e5eb3b2/dns-node-resolver/0.log"
Jul 27 21:46:32.233730 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.233688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs" event={"ID":"7f7d3091-25db-4808-ad26-da8555fdccf3","Type":"ContainerStarted","Data":"cbde7f77ea706a0de951ac4b7270d9be13dc592c970c48df5db8fc36cd46e9fd"}
Jul 27 21:46:32.234065 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.234042    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:32.239707 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.239680    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs"
Jul 27 21:46:32.251531 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.251441    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-m9mqs" podStartSLOduration=1.986260256 podStartE2EDuration="3.251426399s" totalImagesPullingTime="1.265166143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:32.250641545 +0000 UTC m=+57.904070188" watchObservedRunningTime="2026-07-27 21:46:32.251426399 +0000 UTC m=+57.904855042"
Jul 27 21:46:32.586052 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.586020    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:46:32.587975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.587951    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.596326 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.596304    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 21:46:32.598244 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.598221    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:46:32.637904 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.637863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638068 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.637921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638068 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.637966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfkx\" (UniqueName: \"kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638068 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.638040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638233 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.638073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638233 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.638147    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.638233 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.638171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739621 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739621 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739880 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739880 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739709    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739880 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739880 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.739880 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.739789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfkx\" (UniqueName: \"kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.740685 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.740424    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.740685 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.740646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.740837 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.740742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.741228 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.741201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.742834 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.742809    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.742834 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.742828    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.749124 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.749092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfkx\" (UniqueName: \"kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx\") pod \"console-5bcf5f57c-pgs4r\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") " pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.898754 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.898661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:32.969875 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:32.969845    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ts9l5_70ec5f21-87aa-453b-8b4b-999b7484f0b1/node-ca/0.log"
Jul 27 21:46:33.244172 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.244094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:33.244172 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.244143    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:33.244347 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.244218    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:33.244347 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.244263    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:33.244347 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.244281    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.244266291 +0000 UTC m=+62.897694911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:33.244347 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.244303    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:37.244292172 +0000 UTC m=+62.897720792 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:33.394341 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.394312    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"]
Jul 27 21:46:33.403820 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.403791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.407640    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.407794    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.407860    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.408079    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.408178    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 21:46:33.408405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.408255    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8rnfk\""
Jul 27 21:46:33.408855 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.408819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"]
Jul 27 21:46:33.445933 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.445899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.445933 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.445940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74q27\" (UniqueName: \"kubernetes.io/projected/203743aa-5c3d-48a9-a4da-19ef0e94bc20-kube-api-access-74q27\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.446163 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.445975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203743aa-5c3d-48a9-a4da-19ef0e94bc20-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.446163 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.446061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.547516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.547414    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.547516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.547478    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74q27\" (UniqueName: \"kubernetes.io/projected/203743aa-5c3d-48a9-a4da-19ef0e94bc20-kube-api-access-74q27\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.547516 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.547507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203743aa-5c3d-48a9-a4da-19ef0e94bc20-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.547781 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.547558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.547781 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.547767    2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 27 21:46:33.547927 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:33.547832    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls podName:203743aa-5c3d-48a9-a4da-19ef0e94bc20 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:34.047813472 +0000 UTC m=+59.701242114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-8n6vb" (UID: "203743aa-5c3d-48a9-a4da-19ef0e94bc20") : secret "prometheus-operator-tls" not found
Jul 27 21:46:33.548277 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.548213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/203743aa-5c3d-48a9-a4da-19ef0e94bc20-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.550195 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.550172    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.558507 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.558480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74q27\" (UniqueName: \"kubernetes.io/projected/203743aa-5c3d-48a9-a4da-19ef0e94bc20-kube-api-access-74q27\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:33.570626 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.570599    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tj4z9_7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca/serve-healthcheck-canary/0.log"
Jul 27 21:46:33.821218 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:33.818095    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:46:33.821959 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:33.821915    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662f9e21_0c25_451e_b6ff_d63dba87bd0e.slice/crio-2f96b2a9633d4ba5d1f1f76fa470271dd336fe03f4aa824f5fcc5aed24ca8264 WatchSource:0}: Error finding container 2f96b2a9633d4ba5d1f1f76fa470271dd336fe03f4aa824f5fcc5aed24ca8264: Status 404 returned error can't find the container with id 2f96b2a9633d4ba5d1f1f76fa470271dd336fe03f4aa824f5fcc5aed24ca8264
Jul 27 21:46:34.052251 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.052222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:34.054758 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.054735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/203743aa-5c3d-48a9-a4da-19ef0e94bc20-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-8n6vb\" (UID: \"203743aa-5c3d-48a9-a4da-19ef0e94bc20\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:34.240694 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.240657    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf5f57c-pgs4r" event={"ID":"662f9e21-0c25-451e-b6ff-d63dba87bd0e","Type":"ContainerStarted","Data":"2f96b2a9633d4ba5d1f1f76fa470271dd336fe03f4aa824f5fcc5aed24ca8264"}
Jul 27 21:46:34.242097 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.242060    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf788c7df-hflhf" event={"ID":"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55","Type":"ContainerStarted","Data":"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"}
Jul 27 21:46:34.261860 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.261795    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cf788c7df-hflhf" podStartSLOduration=1.671263427 podStartE2EDuration="5.26177931s" totalImagesPullingTime="3.590515883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:34.26098766 +0000 UTC m=+59.914416303" watchObservedRunningTime="2026-07-27 21:46:34.26177931 +0000 UTC m=+59.915207952"
Jul 27 21:46:34.315335 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.315306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"
Jul 27 21:46:34.468433 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:34.468382    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-8n6vb"]
Jul 27 21:46:34.471760 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:34.471737    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203743aa_5c3d_48a9_a4da_19ef0e94bc20.slice/crio-f9402cec848f7dbfd395e0b86dd1762f941348923b5f6af7dec09348173d06d9 WatchSource:0}: Error finding container f9402cec848f7dbfd395e0b86dd1762f941348923b5f6af7dec09348173d06d9: Status 404 returned error can't find the container with id f9402cec848f7dbfd395e0b86dd1762f941348923b5f6af7dec09348173d06d9
Jul 27 21:46:35.247760 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:35.247717    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf5f57c-pgs4r" event={"ID":"662f9e21-0c25-451e-b6ff-d63dba87bd0e","Type":"ContainerStarted","Data":"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"}
Jul 27 21:46:35.249218 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:35.249169    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb" event={"ID":"203743aa-5c3d-48a9-a4da-19ef0e94bc20","Type":"ContainerStarted","Data":"f9402cec848f7dbfd395e0b86dd1762f941348923b5f6af7dec09348173d06d9"}
Jul 27 21:46:35.269750 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:35.269697    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bcf5f57c-pgs4r" podStartSLOduration=2.8130629110000003 podStartE2EDuration="3.269681509s" totalImagesPullingTime="456.618598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:35.26953197 +0000 UTC m=+60.922960614" watchObservedRunningTime="2026-07-27 21:46:35.269681509 +0000 UTC m=+60.923110196"
Jul 27 21:46:36.159785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:36.159713    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bmw8w"
Jul 27 21:46:36.254311 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:36.254277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb" event={"ID":"203743aa-5c3d-48a9-a4da-19ef0e94bc20","Type":"ContainerStarted","Data":"50d7af4fba79c8ee556c83dde438f2fec202a393bd83bb39bc8c56f48588f5d8"}
Jul 27 21:46:36.254732 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:36.254336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb" event={"ID":"203743aa-5c3d-48a9-a4da-19ef0e94bc20","Type":"ContainerStarted","Data":"d6a625d3deb2981fd119277567404c75bdae703ffa18b8ab678a5797121e188c"}
Jul 27 21:46:36.276760 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:36.276711    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-8n6vb" podStartSLOduration=1.9242500900000001 podStartE2EDuration="3.27669309s" totalImagesPullingTime="1.352443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:36.275969339 +0000 UTC m=+61.929397981" watchObservedRunningTime="2026-07-27 21:46:36.27669309 +0000 UTC m=+61.930121734"
Jul 27 21:46:37.279658 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.279617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:37.280086 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.279690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:37.280086 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.279772    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 21:46:37.280086 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.279781    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:37.280086 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.279838    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.279820998 +0000 UTC m=+70.933249618 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : configmap "kube-rbac-proxy" not found
Jul 27 21:46:37.280086 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.279853    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls podName:71c0cb96-1bc0-411c-88bf-2137aae6a264 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:45.279847213 +0000 UTC m=+70.933275833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qm5vp" (UID: "71c0cb96-1bc0-411c-88bf-2137aae6a264") : secret "insights-runtime-extractor-tls" not found
Jul 27 21:46:37.745814 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.745784    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"]
Jul 27 21:46:37.748941 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.748922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.751635 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.751617    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 27 21:46:37.751722 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.751701    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:46:37.751784 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.751707    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hcsdk\""
Jul 27 21:46:37.759636 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.759608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"]
Jul 27 21:46:37.762925 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.762898    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"]
Jul 27 21:46:37.765995 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.765973    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8nt2p"]
Jul 27 21:46:37.766101 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.766079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.768757 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.768738    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 27 21:46:37.768861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.768842    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xzjbc\""
Jul 27 21:46:37.768953 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.768826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.769114 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.769096    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 27 21:46:37.769230 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.768839    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 27 21:46:37.771699 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.771678    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 21:46:37.771796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.771702    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xtsgm\""
Jul 27 21:46:37.771796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.771702    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 21:46:37.771796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.771728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 21:46:37.780336 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.780313    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"]
Jul 27 21:46:37.784390 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.784368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.784491 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.784396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7455e-cd15-4f81-97bc-d8a0b6eab446-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.784491 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.784442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmv6z\" (UniqueName: \"kubernetes.io/projected/41e7455e-cd15-4f81-97bc-d8a0b6eab446-kube-api-access-mmv6z\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.784491 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.784481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.885045 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-root\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885045 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-metrics-client-ca\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7455e-cd15-4f81-97bc-d8a0b6eab446-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-sys\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885290 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885268    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885365    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmv6z\" (UniqueName: \"kubernetes.io/projected/41e7455e-cd15-4f81-97bc-d8a0b6eab446-kube-api-access-mmv6z\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwmhr\" (UniqueName: \"kubernetes.io/projected/670d8499-96e3-452a-8545-8f8bd041e369-kube-api-access-vwmhr\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-textfile\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-wtmp\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3faccd65-c981-4247-971e-1e39178a6710-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsfb\" (UniqueName: \"kubernetes.io/projected/3faccd65-c981-4247-971e-1e39178a6710-kube-api-access-2gsfb\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.885912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.885912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.885882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41e7455e-cd15-4f81-97bc-d8a0b6eab446-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.887664 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.887634    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.887885 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.887868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/41e7455e-cd15-4f81-97bc-d8a0b6eab446-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.894790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.894771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmv6z\" (UniqueName: \"kubernetes.io/projected/41e7455e-cd15-4f81-97bc-d8a0b6eab446-kube-api-access-mmv6z\") pod \"openshift-state-metrics-b8f49f5c6-ml84k\" (UID: \"41e7455e-cd15-4f81-97bc-d8a0b6eab446\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:37.986393 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-root\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986582 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986399    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-metrics-client-ca\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986582 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-root\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986582 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.986582 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-sys\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-sys\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986712    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwmhr\" (UniqueName: \"kubernetes.io/projected/670d8499-96e3-452a-8545-8f8bd041e369-kube-api-access-vwmhr\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986738    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-textfile\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.986803 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-wtmp\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3faccd65-c981-4247-971e-1e39178a6710-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986845    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsfb\" (UniqueName: \"kubernetes.io/projected/3faccd65-c981-4247-971e-1e39178a6710-kube-api-access-2gsfb\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.986952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-wtmp\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-metrics-client-ca\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987485 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-textfile\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987485 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.987293    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 21:46:37.987485 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987324    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-accelerators-collector-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.987485 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:37.987359    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls podName:670d8499-96e3-452a-8545-8f8bd041e369 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:38.487341457 +0000 UTC m=+64.140770089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls") pod "node-exporter-8nt2p" (UID: "670d8499-96e3-452a-8545-8f8bd041e369") : secret "node-exporter-tls" not found
Jul 27 21:46:37.987858 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987513    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3faccd65-c981-4247-971e-1e39178a6710-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.987858 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.987969 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.987896    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.989279 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.989259    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.989426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.989405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3faccd65-c981-4247-971e-1e39178a6710-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.989554 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.989539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:37.995688 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.995666    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsfb\" (UniqueName: \"kubernetes.io/projected/3faccd65-c981-4247-971e-1e39178a6710-kube-api-access-2gsfb\") pod \"kube-state-metrics-56d9f944b4-8cxmp\" (UID: \"3faccd65-c981-4247-971e-1e39178a6710\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:37.996018 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:37.995979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwmhr\" (UniqueName: \"kubernetes.io/projected/670d8499-96e3-452a-8545-8f8bd041e369-kube-api-access-vwmhr\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:38.057610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.057579    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"
Jul 27 21:46:38.078084 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.078060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"
Jul 27 21:46:38.195571 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.195544    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k"]
Jul 27 21:46:38.197996 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:38.197967    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e7455e_cd15_4f81_97bc_d8a0b6eab446.slice/crio-624964af5dd2e0d5e87ecfc2a29a944f9e25489c397f55c9651f3dc780119a14 WatchSource:0}: Error finding container 624964af5dd2e0d5e87ecfc2a29a944f9e25489c397f55c9651f3dc780119a14: Status 404 returned error can't find the container with id 624964af5dd2e0d5e87ecfc2a29a944f9e25489c397f55c9651f3dc780119a14
Jul 27 21:46:38.210069 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.210048    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp"]
Jul 27 21:46:38.213830 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:38.213806    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faccd65_c981_4247_971e_1e39178a6710.slice/crio-ac5770a54f550248eb4ecbf1255852bed403bfdc7adad53b83bcd639f0ae5930 WatchSource:0}: Error finding container ac5770a54f550248eb4ecbf1255852bed403bfdc7adad53b83bcd639f0ae5930: Status 404 returned error can't find the container with id ac5770a54f550248eb4ecbf1255852bed403bfdc7adad53b83bcd639f0ae5930
Jul 27 21:46:38.221718 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.221697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8m9h8"
Jul 27 21:46:38.271804 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.271772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp" event={"ID":"3faccd65-c981-4247-971e-1e39178a6710","Type":"ContainerStarted","Data":"ac5770a54f550248eb4ecbf1255852bed403bfdc7adad53b83bcd639f0ae5930"}
Jul 27 21:46:38.273527 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.273501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k" event={"ID":"41e7455e-cd15-4f81-97bc-d8a0b6eab446","Type":"ContainerStarted","Data":"3ed2d0e6220a33ed0cfe88f781915c08111c0653894e4be708c28609e7273f00"}
Jul 27 21:46:38.273628 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.273558    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k" event={"ID":"41e7455e-cd15-4f81-97bc-d8a0b6eab446","Type":"ContainerStarted","Data":"624964af5dd2e0d5e87ecfc2a29a944f9e25489c397f55c9651f3dc780119a14"}
Jul 27 21:46:38.492355 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.492314    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:38.494813 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.494795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/670d8499-96e3-452a-8545-8f8bd041e369-node-exporter-tls\") pod \"node-exporter-8nt2p\" (UID: \"670d8499-96e3-452a-8545-8f8bd041e369\") " pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:38.684767 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.684717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8nt2p"
Jul 27 21:46:38.695091 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:38.695055    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670d8499_96e3_452a_8545_8f8bd041e369.slice/crio-707a5f38fa494a8f68d3bb03041c633a387ff38b86d95d8d8dc62b35409ce142 WatchSource:0}: Error finding container 707a5f38fa494a8f68d3bb03041c633a387ff38b86d95d8d8dc62b35409ce142: Status 404 returned error can't find the container with id 707a5f38fa494a8f68d3bb03041c633a387ff38b86d95d8d8dc62b35409ce142
Jul 27 21:46:38.838974 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.838943    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:46:38.843690 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.843670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.848073 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848010    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:46:38.848073 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848064    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:46:38.848512 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:46:38.848648 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:46:38.848648 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848494    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:46:38.848782 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.848753    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:46:38.849057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.849021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:46:38.849165 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.849076    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kgq96\""
Jul 27 21:46:38.849165 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.849027    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:46:38.849165 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.849036    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:46:38.859695 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.859667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:46:38.895624 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.895790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.895790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.895790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.895790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.895790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sj4\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895859    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.895975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.896005    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.896040 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.896029    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997282 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997282 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997282 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997629 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997518    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997629 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997559    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997629 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997597    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997627    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997681    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sj4\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.997796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.998284 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.997937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:38.998284 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:46:38.998065    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle podName:509f17ca-db52-4b93-bd5e-eb81f3cbb570 nodeName:}" failed. No retries permitted until 2026-07-27 21:46:39.498046246 +0000 UTC m=+65.151474881 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570") : configmap references non-existent config key: ca-bundle.crt
Jul 27 21:46:38.999093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:38.999067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.001010 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.000984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.003195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.003638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.003791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.003897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.004072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.004526 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.004173    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.005235 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.005194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.005235 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.005214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.009282 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.009258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sj4\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.279382 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.279289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k" event={"ID":"41e7455e-cd15-4f81-97bc-d8a0b6eab446","Type":"ContainerStarted","Data":"488aff983ea48fef85668df938730e12ec10ee4240045c09cf63e90b72e075c7"}
Jul 27 21:46:39.280642 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.280606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nt2p" event={"ID":"670d8499-96e3-452a-8545-8f8bd041e369","Type":"ContainerStarted","Data":"707a5f38fa494a8f68d3bb03041c633a387ff38b86d95d8d8dc62b35409ce142"}
Jul 27 21:46:39.502134 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.502081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.503123 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.503092    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:39.754890 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:39.754856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:46:40.165566 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.165536    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:46:40.170366 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:40.170322    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509f17ca_db52_4b93_bd5e_eb81f3cbb570.slice/crio-5530b1f09cf6401ddfc5936367cfeea15815c06f22174c2716cc7bc42c089e8c WatchSource:0}: Error finding container 5530b1f09cf6401ddfc5936367cfeea15815c06f22174c2716cc7bc42c089e8c: Status 404 returned error can't find the container with id 5530b1f09cf6401ddfc5936367cfeea15815c06f22174c2716cc7bc42c089e8c
Jul 27 21:46:40.286863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.286754    2560 generic.go:356] "Generic (PLEG): container finished" podID="670d8499-96e3-452a-8545-8f8bd041e369" containerID="bab8798cfb5fffe502014804b97a05537969aab91b41e3dc97bb4b85e815652c" exitCode=0
Jul 27 21:46:40.287038 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.286875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nt2p" event={"ID":"670d8499-96e3-452a-8545-8f8bd041e369","Type":"ContainerDied","Data":"bab8798cfb5fffe502014804b97a05537969aab91b41e3dc97bb4b85e815652c"}
Jul 27 21:46:40.290371 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.290343    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp" event={"ID":"3faccd65-c981-4247-971e-1e39178a6710","Type":"ContainerStarted","Data":"e4626b1901d829636ed2977028c89aaa39d720215b407e82f9a3751632860e4f"}
Jul 27 21:46:40.290519 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.290397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp" event={"ID":"3faccd65-c981-4247-971e-1e39178a6710","Type":"ContainerStarted","Data":"53bb6d76848a8856cf04995c20a83d00d0b495c754545c21723111756ea156ed"}
Jul 27 21:46:40.294214 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.292989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k" event={"ID":"41e7455e-cd15-4f81-97bc-d8a0b6eab446","Type":"ContainerStarted","Data":"fbc0a0d1d7cb6ccd38a491f9b2df35c4809ddd12f2b5f7df4861747ddadb82ad"}
Jul 27 21:46:40.295774 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.295741    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"5530b1f09cf6401ddfc5936367cfeea15815c06f22174c2716cc7bc42c089e8c"}
Jul 27 21:46:40.322658 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.322019    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:40.322658 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.322070    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:40.327886 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.327798    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:40.397223 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.397165    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-ml84k" podStartSLOduration=1.707154579 podStartE2EDuration="3.397144395s" totalImagesPullingTime="1.689989816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:40.396691869 +0000 UTC m=+66.050120540" watchObservedRunningTime="2026-07-27 21:46:40.397144395 +0000 UTC m=+66.050573038"
Jul 27 21:46:40.511416 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.511362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:40.513811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.513791    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 21:46:40.524092 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.524066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/800e44b7-5b8c-4c51-af1d-31631bcc2306-metrics-certs\") pod \"network-metrics-daemon-qbjt6\" (UID: \"800e44b7-5b8c-4c51-af1d-31631bcc2306\") " pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:40.612648 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.612613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:40.615384 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.615178    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 21:46:40.625670 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.625643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6fe41b7e-2a93-4659-923b-abd00fdabeec-original-pull-secret\") pod \"global-pull-secret-syncer-7rm99\" (UID: \"6fe41b7e-2a93-4659-923b-abd00fdabeec\") " pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:40.629558 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.629535    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7rm99"
Jul 27 21:46:40.713265 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.713230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:40.716260 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.715999    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 21:46:40.726211 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.726183    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 21:46:40.737047 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.736984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bp5\" (UniqueName: \"kubernetes.io/projected/2ba9ea4a-bac1-44da-b1af-052c8d54f737-kube-api-access-65bp5\") pod \"network-check-target-qlsm7\" (UID: \"2ba9ea4a-bac1-44da-b1af-052c8d54f737\") " pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:40.749863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.749827    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"]
Jul 27 21:46:40.772987 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.772955    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"]
Jul 27 21:46:40.773170 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.773066    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.775528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.775500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 27 21:46:40.775528 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.775521    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 27 21:46:40.776110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.776088    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1cjq8hgkb9i0j\""
Jul 27 21:46:40.776230 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.776186    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5fd8k\""
Jul 27 21:46:40.776230 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.776219    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 27 21:46:40.776357 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.776303    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 27 21:46:40.776415 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.776361    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 27 21:46:40.780914 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.780859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7rm99"]
Jul 27 21:46:40.786354 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:40.786327    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe41b7e_2a93_4659_923b_abd00fdabeec.slice/crio-ca0a6f39ed7db6cb633645e8001e564316ceaee93d8e495a7b40d631595c8cb5 WatchSource:0}: Error finding container ca0a6f39ed7db6cb633645e8001e564316ceaee93d8e495a7b40d631595c8cb5: Status 404 returned error can't find the container with id ca0a6f39ed7db6cb633645e8001e564316ceaee93d8e495a7b40d631595c8cb5
Jul 27 21:46:40.814146 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814113    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814308 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-grpc-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814308 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814193    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814308 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814286    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814486 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8l5d\" (UniqueName: \"kubernetes.io/projected/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-kube-api-access-s8l5d\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814486 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814595 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814507    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-metrics-client-ca\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.814595 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.814580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.825323 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.825286    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cj7gv\""
Jul 27 21:46:40.833497 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.833446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qbjt6"
Jul 27 21:46:40.918302 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918538 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918510    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-metrics-client-ca\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918689 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918599    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918689 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918689 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-grpc-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918847 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.918847 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.918750    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.919585 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.919008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8l5d\" (UniqueName: \"kubernetes.io/projected/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-kube-api-access-s8l5d\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.919930 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.919905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-metrics-client-ca\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.923155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.923103    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.923884 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.923807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.923884 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.923840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-grpc-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.924265 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.924247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-tls\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.924521 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.924494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.924967 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.924949    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kt796\""
Jul 27 21:46:40.925694 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.925661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.931168 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.931143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8l5d\" (UniqueName: \"kubernetes.io/projected/b3e4b5db-eba5-4147-8a8b-6a3badceea0d-kube-api-access-s8l5d\") pod \"thanos-querier-6ddd7657b4-qcvcq\" (UID: \"b3e4b5db-eba5-4147-8a8b-6a3badceea0d\") " pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:40.933900 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.933881    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:40.998939 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:40.998865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qbjt6"]
Jul 27 21:46:41.004410 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:41.004378    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800e44b7_5b8c_4c51_af1d_31631bcc2306.slice/crio-07c80708d1732dad0a5183e9649e4005c0073c518f7d4db47676440ac5eb4c1a WatchSource:0}: Error finding container 07c80708d1732dad0a5183e9649e4005c0073c518f7d4db47676440ac5eb4c1a: Status 404 returned error can't find the container with id 07c80708d1732dad0a5183e9649e4005c0073c518f7d4db47676440ac5eb4c1a
Jul 27 21:46:41.086952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.086916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:41.099026 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.098998    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qlsm7"]
Jul 27 21:46:41.302556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.302469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nt2p" event={"ID":"670d8499-96e3-452a-8545-8f8bd041e369","Type":"ContainerStarted","Data":"746aec8c074fe9aa078ca41f2729bd61119fdfb0525df5db307eee881810d5de"}
Jul 27 21:46:41.302556 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.302532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8nt2p" event={"ID":"670d8499-96e3-452a-8545-8f8bd041e369","Type":"ContainerStarted","Data":"f2c7b3c4b9807bb753efd6b366a3159301b9e1df48cc5f55dafad1041e2566d1"}
Jul 27 21:46:41.303987 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.303928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbjt6" event={"ID":"800e44b7-5b8c-4c51-af1d-31631bcc2306","Type":"ContainerStarted","Data":"07c80708d1732dad0a5183e9649e4005c0073c518f7d4db47676440ac5eb4c1a"}
Jul 27 21:46:41.306381 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.306351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp" event={"ID":"3faccd65-c981-4247-971e-1e39178a6710","Type":"ContainerStarted","Data":"8305b44b56b07a531b1a845c5caaf3807229dbeb3f6a57adffc4c3bc6df093bc"}
Jul 27 21:46:41.307694 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.307669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7rm99" event={"ID":"6fe41b7e-2a93-4659-923b-abd00fdabeec","Type":"ContainerStarted","Data":"ca0a6f39ed7db6cb633645e8001e564316ceaee93d8e495a7b40d631595c8cb5"}
Jul 27 21:46:41.312120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.312097    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:46:41.322023 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.321975    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8nt2p" podStartSLOduration=2.852937297 podStartE2EDuration="4.321957063s" totalImagesPullingTime="1.361228647s" totalInitContainerRuntime="107.791119ms" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:41.320219794 +0000 UTC m=+66.973648439" watchObservedRunningTime="2026-07-27 21:46:41.321957063 +0000 UTC m=+66.975385706"
Jul 27 21:46:41.358202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.358140    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-8cxmp" podStartSLOduration=2.558515737 podStartE2EDuration="4.358119056s" totalImagesPullingTime="1.799603319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:41.356704689 +0000 UTC m=+67.010133372" watchObservedRunningTime="2026-07-27 21:46:41.358119056 +0000 UTC m=+67.011547699"
Jul 27 21:46:41.410201 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:41.410160    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba9ea4a_bac1_44da_b1af_052c8d54f737.slice/crio-7210bdde3e5e20d20b355c34e300e985cba19c72f6a741f8be8f8183be64096b WatchSource:0}: Error finding container 7210bdde3e5e20d20b355c34e300e985cba19c72f6a741f8be8f8183be64096b: Status 404 returned error can't find the container with id 7210bdde3e5e20d20b355c34e300e985cba19c72f6a741f8be8f8183be64096b
Jul 27 21:46:41.547699 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:41.547663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"]
Jul 27 21:46:41.551899 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:41.551871    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e4b5db_eba5_4147_8a8b_6a3badceea0d.slice/crio-871dea3468355a4c6e2c76f9f70606295ec305620e23d38428656ae5e5519637 WatchSource:0}: Error finding container 871dea3468355a4c6e2c76f9f70606295ec305620e23d38428656ae5e5519637: Status 404 returned error can't find the container with id 871dea3468355a4c6e2c76f9f70606295ec305620e23d38428656ae5e5519637
Jul 27 21:46:42.076815 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.076739    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7b669bc878-mxn88"]
Jul 27 21:46:42.101110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.101051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b669bc878-mxn88"]
Jul 27 21:46:42.101412 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.101190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.104239 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.104008    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 27 21:46:42.104720 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.104550    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 27 21:46:42.105703 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.105089    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 27 21:46:42.105703 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.105109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-g9kqg\""
Jul 27 21:46:42.105703 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.105324    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 27 21:46:42.105703 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.105519    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-duta9hula459\""
Jul 27 21:46:42.232284 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ml4\" (UniqueName: \"kubernetes.io/projected/897af678-0db9-4f38-816e-2bd664b91361-kube-api-access-z4ml4\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232473 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232303    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-client-certs\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232473 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/897af678-0db9-4f38-816e-2bd664b91361-audit-log\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232473 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-tls\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232663 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232538    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-metrics-server-audit-profiles\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232663 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-client-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.232663 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.232606    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.314962 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.314921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"871dea3468355a4c6e2c76f9f70606295ec305620e23d38428656ae5e5519637"}
Jul 27 21:46:42.316877 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.316848    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qlsm7" event={"ID":"2ba9ea4a-bac1-44da-b1af-052c8d54f737","Type":"ContainerStarted","Data":"7210bdde3e5e20d20b355c34e300e985cba19c72f6a741f8be8f8183be64096b"}
Jul 27 21:46:42.319103 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.319064    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24" exitCode=0
Jul 27 21:46:42.321265 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.319213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"}
Jul 27 21:46:42.334063 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.333992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ml4\" (UniqueName: \"kubernetes.io/projected/897af678-0db9-4f38-816e-2bd664b91361-kube-api-access-z4ml4\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334063 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-client-certs\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/897af678-0db9-4f38-816e-2bd664b91361-audit-log\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-tls\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334176    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-metrics-server-audit-profiles\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-client-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.334292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.334240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.335421 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.335395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.335708 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.335685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/897af678-0db9-4f38-816e-2bd664b91361-audit-log\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.336534 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.336509    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/897af678-0db9-4f38-816e-2bd664b91361-metrics-server-audit-profiles\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.340287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.340245    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-tls\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.340732 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.340635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-secret-metrics-server-client-certs\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.341699 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.341654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/897af678-0db9-4f38-816e-2bd664b91361-client-ca-bundle\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.343800 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.343758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ml4\" (UniqueName: \"kubernetes.io/projected/897af678-0db9-4f38-816e-2bd664b91361-kube-api-access-z4ml4\") pod \"metrics-server-7b669bc878-mxn88\" (UID: \"897af678-0db9-4f38-816e-2bd664b91361\") " pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.416765 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.416562    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:46:42.553204 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.553174    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"]
Jul 27 21:46:42.558084 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.557573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.568823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.560952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 27 21:46:42.568823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.568834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"]
Jul 27 21:46:42.569507 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.569111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-m9m5z\""
Jul 27 21:46:42.619158 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.619103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7b669bc878-mxn88"]
Jul 27 21:46:42.622790 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:42.622740    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897af678_0db9_4f38_816e_2bd664b91361.slice/crio-c31ac4560475bda5c4496c210558141b08695d7a081a576e7bf868c6b20e30f7 WatchSource:0}: Error finding container c31ac4560475bda5c4496c210558141b08695d7a081a576e7bf868c6b20e30f7: Status 404 returned error can't find the container with id c31ac4560475bda5c4496c210558141b08695d7a081a576e7bf868c6b20e30f7
Jul 27 21:46:42.637427 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.637197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.637427 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.637254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzlm9\" (UniqueName: \"kubernetes.io/projected/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-kube-api-access-dzlm9\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.739740 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.738755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.739740 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.738822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzlm9\" (UniqueName: \"kubernetes.io/projected/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-kube-api-access-dzlm9\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.742931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.742886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.749122 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.749098    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzlm9\" (UniqueName: \"kubernetes.io/projected/4d8ad193-3e0b-4f14-a17c-685c9c034e1b-kube-api-access-dzlm9\") pod \"monitoring-plugin-68b7fc9574-f5mzs\" (UID: \"4d8ad193-3e0b-4f14-a17c-685c9c034e1b\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.872818 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.872523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:42.900219 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.899436    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:42.900219 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.900185    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:42.906725 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.906513    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:42.962589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.962559    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"]
Jul 27 21:46:42.968712 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.967993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:42.970576 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.970489    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 27 21:46:42.979910 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.979869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 27 21:46:42.985727 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.983297    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vwnsd\""
Jul 27 21:46:42.985727 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.983621    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 27 21:46:42.985727 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.984058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 27 21:46:42.985948 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.985786    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 27 21:46:42.997203 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:42.997176    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"]
Jul 27 21:46:43.000902 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.000879    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 27 21:46:43.042604 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-serving-certs-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042774 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042632    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-federate-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042774 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042774 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcgw\" (UniqueName: \"kubernetes.io/projected/7dc9b9ca-246a-434f-9756-bfd43de6df05-kube-api-access-8fcgw\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042873    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-metrics-client-ca\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.042943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.042912    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.055121 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.055006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"]
Jul 27 21:46:43.070405 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:43.070373    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8ad193_3e0b_4f14_a17c_685c9c034e1b.slice/crio-f2e1ab99baeb2818914de174e9ee6ca14f700227c347766f9170349f194a7f02 WatchSource:0}: Error finding container f2e1ab99baeb2818914de174e9ee6ca14f700227c347766f9170349f194a7f02: Status 404 returned error can't find the container with id f2e1ab99baeb2818914de174e9ee6ca14f700227c347766f9170349f194a7f02
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.143492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-serving-certs-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.143553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-federate-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.143625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.143669    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcgw\" (UniqueName: \"kubernetes.io/projected/7dc9b9ca-246a-434f-9756-bfd43de6df05-kube-api-access-8fcgw\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.143702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.144495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.144212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.145881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.145050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-metrics-client-ca\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.145881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.145121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.145881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.145127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-serving-certs-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.145881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.145801    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-metrics-client-ca\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.146173 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.146004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.149007 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.148959    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.149672 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.149646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-secret-telemeter-client\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.151583 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.151560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-telemeter-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.151998 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.151974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7dc9b9ca-246a-434f-9756-bfd43de6df05-federate-client-tls\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.154897 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.154871    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcgw\" (UniqueName: \"kubernetes.io/projected/7dc9b9ca-246a-434f-9756-bfd43de6df05-kube-api-access-8fcgw\") pod \"telemeter-client-5586cd6d9b-5qfqs\" (UID: \"7dc9b9ca-246a-434f-9756-bfd43de6df05\") " pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.304868 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.304825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"
Jul 27 21:46:43.330339 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.330244    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs" event={"ID":"4d8ad193-3e0b-4f14-a17c-685c9c034e1b","Type":"ContainerStarted","Data":"f2e1ab99baeb2818914de174e9ee6ca14f700227c347766f9170349f194a7f02"}
Jul 27 21:46:43.333951 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.333910    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88" event={"ID":"897af678-0db9-4f38-816e-2bd664b91361","Type":"ContainerStarted","Data":"c31ac4560475bda5c4496c210558141b08695d7a081a576e7bf868c6b20e30f7"}
Jul 27 21:46:43.349099 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.343783    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbjt6" event={"ID":"800e44b7-5b8c-4c51-af1d-31631bcc2306","Type":"ContainerStarted","Data":"fd2105e0e7e5d7250e4b5b795a0b65b5fc4352e8cdc2c6c945ad07086b5f2064"}
Jul 27 21:46:43.349099 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.343863    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qbjt6" event={"ID":"800e44b7-5b8c-4c51-af1d-31631bcc2306","Type":"ContainerStarted","Data":"2e2d5721f30e758c8dd7c9b5d5ff85ad12533986ab61c87e67ee02b87f9131d2"}
Jul 27 21:46:43.351358 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.350866    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:46:43.395619 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.394465    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qbjt6" podStartSLOduration=66.969609363 podStartE2EDuration="1m8.394425868s" totalImagesPullingTime="1.424816505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:43.368041581 +0000 UTC m=+69.021470223" watchObservedRunningTime="2026-07-27 21:46:43.394425868 +0000 UTC m=+69.047854510"
Jul 27 21:46:43.434157 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.434115    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:46:43.546093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:43.546059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs"]
Jul 27 21:46:43.547700 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:43.547660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc9b9ca_246a_434f_9756_bfd43de6df05.slice/crio-3c6958fbcbfee296dfbff003ba43f05fd0cdccb3dd25641200c5b161bb13ccd3 WatchSource:0}: Error finding container 3c6958fbcbfee296dfbff003ba43f05fd0cdccb3dd25641200c5b161bb13ccd3: Status 404 returned error can't find the container with id 3c6958fbcbfee296dfbff003ba43f05fd0cdccb3dd25641200c5b161bb13ccd3
Jul 27 21:46:44.102610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.102426    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:46:44.108524 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.108504    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.111496    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.112368    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.112526    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.112646    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.112752    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.112884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:46:44.113589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.113427    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:46:44.113984 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.113656    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:46:44.113984 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.113683    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjnbj\""
Jul 27 21:46:44.113984 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.113840    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:46:44.113984 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.113909    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:46:44.116025 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.114817    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-975fml6qnmh2g\""
Jul 27 21:46:44.116025 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.115567    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:46:44.117943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.117740    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:46:44.120599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.119812    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:46:44.123697 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.123676    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:46:44.127174 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.127116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:46:44.162666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.162633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwc42\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.163161 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.163132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.163384 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.163357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.163603 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.163585    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.163834 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.163816    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164048 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164231 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164344 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164474 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164607 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164778    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.164937 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.164920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.165081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165057    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.165286 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165269    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.165416 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165399    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.165615 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165581    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.165835 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.166020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.165980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.166237 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.166219    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.166552 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.166521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268706    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268780    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268872    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268902    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268928    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268952    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.268998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269086    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.269273 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwc42\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.270211 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.270211 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.270211 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.270211 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.269555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.271533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.270530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.271533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.270771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.271533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.271430    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.271901 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.271876    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.272304 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.272279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.273480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.273246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.274938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.274396    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.274938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.274891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.278167 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.278118    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.279387 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.279363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.280133 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.280066    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.281247 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.281221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwc42\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.282299 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.281885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.284261 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.284190    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.284373 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.284294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.285092 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.285005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.286263 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.286241    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.287413 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.287380    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.288069 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.288045    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:44.353442 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.353358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs" event={"ID":"7dc9b9ca-246a-434f-9756-bfd43de6df05","Type":"ContainerStarted","Data":"3c6958fbcbfee296dfbff003ba43f05fd0cdccb3dd25641200c5b161bb13ccd3"}
Jul 27 21:46:44.430031 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:44.429994    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:46:45.280675 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.280629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:45.281105 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.280763    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:45.281238 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.281206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/71c0cb96-1bc0-411c-88bf-2137aae6a264-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:45.283712 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.283676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/71c0cb96-1bc0-411c-88bf-2137aae6a264-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qm5vp\" (UID: \"71c0cb96-1bc0-411c-88bf-2137aae6a264\") " pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:45.469899 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.469864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7smr8\""
Jul 27 21:46:45.477956 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:45.477912    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qm5vp"
Jul 27 21:46:49.098419 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.098375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:46:49.104117 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:49.104087    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83cd7e2_8359_44b4_916c_911fc769cf7d.slice/crio-7839f414aba65ef71c585afd8af92eae7733cdccb0fcae07ce2a6ebc96210322 WatchSource:0}: Error finding container 7839f414aba65ef71c585afd8af92eae7733cdccb0fcae07ce2a6ebc96210322: Status 404 returned error can't find the container with id 7839f414aba65ef71c585afd8af92eae7733cdccb0fcae07ce2a6ebc96210322
Jul 27 21:46:49.128120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.126721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qm5vp"]
Jul 27 21:46:49.137546 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:46:49.136859    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c0cb96_1bc0_411c_88bf_2137aae6a264.slice/crio-c559745fc2cfe148af61062a941b710d24f267a5a38ebda747b7c4d8f9b01a56 WatchSource:0}: Error finding container c559745fc2cfe148af61062a941b710d24f267a5a38ebda747b7c4d8f9b01a56: Status 404 returned error can't find the container with id c559745fc2cfe148af61062a941b710d24f267a5a38ebda747b7c4d8f9b01a56
Jul 27 21:46:49.376055 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.376024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs" event={"ID":"7dc9b9ca-246a-434f-9756-bfd43de6df05","Type":"ContainerStarted","Data":"1449feb5ba516388784980d121127b7539e844f5dd0c51106c7c7bee40ecd131"}
Jul 27 21:46:49.376207 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.376082    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs" event={"ID":"7dc9b9ca-246a-434f-9756-bfd43de6df05","Type":"ContainerStarted","Data":"8b6c4a58d453976070f329b95bca0fda341b295244ccd82716edc2a152e1cdbc"}
Jul 27 21:46:49.376207 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.376113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs" event={"ID":"7dc9b9ca-246a-434f-9756-bfd43de6df05","Type":"ContainerStarted","Data":"1f71c928ca0a3e04d9100048c69a05845602c9f88cc8e38b1b1e9961ab7d1984"}
Jul 27 21:46:49.385371 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.385339    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88" event={"ID":"897af678-0db9-4f38-816e-2bd664b91361","Type":"ContainerStarted","Data":"711d81c7a539d6434930d9d15643b7dc1735ca42ae091135640c84ae596f5ee7"}
Jul 27 21:46:49.387142 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.387113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7rm99" event={"ID":"6fe41b7e-2a93-4659-923b-abd00fdabeec","Type":"ContainerStarted","Data":"c7edde23875c820d2ad72ab67f36f01444d2dee00b33d35de5894bbdca08165d"}
Jul 27 21:46:49.389172 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.389148    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"605f2c1e59848440f1fa38d56d87d9a6f7679e30ce2039fd9d6c7b0735d24ff4"}
Jul 27 21:46:49.389266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.389214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"f1f26238324d294cc3c6df7a7e41757a3d0ba96eb568ccd1d211d9aa7ce9d47b"}
Jul 27 21:46:49.389266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.389254    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"1066ebaeef2bef047195ce01ef0b76ca2a636acd4124ca3938b61b27eb396cda"}
Jul 27 21:46:49.392002 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.391972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"}
Jul 27 21:46:49.392112 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.392040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"}
Jul 27 21:46:49.392112 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.392080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"}
Jul 27 21:46:49.393945 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.393922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qm5vp" event={"ID":"71c0cb96-1bc0-411c-88bf-2137aae6a264","Type":"ContainerStarted","Data":"c559745fc2cfe148af61062a941b710d24f267a5a38ebda747b7c4d8f9b01a56"}
Jul 27 21:46:49.395755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.395733    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c" exitCode=0
Jul 27 21:46:49.395913 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.395777    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"}
Jul 27 21:46:49.396217 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.396199    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"7839f414aba65ef71c585afd8af92eae7733cdccb0fcae07ce2a6ebc96210322"}
Jul 27 21:46:49.398875 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.398851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qlsm7" event={"ID":"2ba9ea4a-bac1-44da-b1af-052c8d54f737","Type":"ContainerStarted","Data":"0027a2e6bf955a0ed70dde5de813b1e9330cc6bb4b048d1c37b85cf48aa6f419"}
Jul 27 21:46:49.399086 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.399068    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:46:49.400171 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.400084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5586cd6d9b-5qfqs" podStartSLOduration=1.960091261 podStartE2EDuration="7.400064513s" totalImagesPullingTime="5.439973252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:49.396106082 +0000 UTC m=+75.049534747" watchObservedRunningTime="2026-07-27 21:46:49.400064513 +0000 UTC m=+75.053493161"
Jul 27 21:46:49.400613 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.400592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs" event={"ID":"4d8ad193-3e0b-4f14-a17c-685c9c034e1b","Type":"ContainerStarted","Data":"12aa9e4c4c74bc7cd91b26291ebadf2deb78728632c15e54860bc4ee2741ba16"}
Jul 27 21:46:49.400814 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.400790    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:49.406796 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.406747    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs"
Jul 27 21:46:49.414538 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.414493    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88" podStartSLOduration=1.131496349 podStartE2EDuration="7.41447645s" totalImagesPullingTime="6.282980101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:49.41271845 +0000 UTC m=+75.066147103" watchObservedRunningTime="2026-07-27 21:46:49.41447645 +0000 UTC m=+75.067905092"
Jul 27 21:46:49.461577 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.461528    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7rm99" podStartSLOduration=66.341724498 podStartE2EDuration="1m14.461511455s" totalImagesPullingTime="8.119786957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:49.460414831 +0000 UTC m=+75.113843474" watchObservedRunningTime="2026-07-27 21:46:49.461511455 +0000 UTC m=+75.114940087"
Jul 27 21:46:49.478015 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.477962    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qlsm7" podStartSLOduration=66.981010802 podStartE2EDuration="1m14.477942415s" totalImagesPullingTime="7.496931613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:49.477013757 +0000 UTC m=+75.130442402" watchObservedRunningTime="2026-07-27 21:46:49.477942415 +0000 UTC m=+75.131371055"
Jul 27 21:46:49.495234 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:49.495166    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-f5mzs" podStartSLOduration=1.659604398 podStartE2EDuration="7.495143797s" totalImagesPullingTime="5.835539399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:49.493982086 +0000 UTC m=+75.147410729" watchObservedRunningTime="2026-07-27 21:46:49.495143797 +0000 UTC m=+75.148572440"
Jul 27 21:46:50.412391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:50.412355    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"}
Jul 27 21:46:50.412889 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:50.412436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"}
Jul 27 21:46:51.307141 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.307103    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:46:51.312298 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.312093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.325150 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.325085    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:46:51.424180 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.424133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"14d9b154d8af5862a116acf3dbe7555d78742470bb11f6eee66a870350a29b9e"}
Jul 27 21:46:51.424705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.424221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"ae220bba72aeed85037db1b571956d3e950e9124ff0e39df6ec750ec4b185fc5"}
Jul 27 21:46:51.424705 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.424262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" event={"ID":"b3e4b5db-eba5-4147-8a8b-6a3badceea0d","Type":"ContainerStarted","Data":"45807f2fafa261e66d07ee3e2f5bf9bfe25b5ec47b195fc22561e14ea063ce2f"}
Jul 27 21:46:51.425353 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.425329    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:51.430743 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.430662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerStarted","Data":"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"}
Jul 27 21:46:51.451926 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.451891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.451948    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7vc\" (UniqueName: \"kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.452022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.452050    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.452065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.452098    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.452287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.452126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.478354 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.476558    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq" podStartSLOduration=2.83223366 podStartE2EDuration="11.476539845s" totalImagesPullingTime="8.644306185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:51.451362044 +0000 UTC m=+77.104790696" watchObservedRunningTime="2026-07-27 21:46:51.476539845 +0000 UTC m=+77.129968515"
Jul 27 21:46:51.478354 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.477000    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.139465519 podStartE2EDuration="13.476988025s" totalImagesPullingTime="9.304394059s" totalInitContainerRuntime="33.128447ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:51.475376489 +0000 UTC m=+77.128805131" watchObservedRunningTime="2026-07-27 21:46:51.476988025 +0000 UTC m=+77.130416670"
Jul 27 21:46:51.553097 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553269 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7vc\" (UniqueName: \"kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553339 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553304    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553392 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553392 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553524 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.553580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.553568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.556485 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.554963    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.556485 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.555965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.556840 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.556819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.557629 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.557571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.560307 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.560281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.564986 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.564923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7vc\" (UniqueName: \"kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.565397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.565374    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert\") pod \"console-5759cb7889-qrq2v\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") " pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.628678 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.628641    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:46:51.805308 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:51.805217    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:46:52.439601 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:52.439560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5759cb7889-qrq2v" event={"ID":"0e52e651-27a3-4c53-b525-066b26f3a67e","Type":"ContainerStarted","Data":"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"}
Jul 27 21:46:52.439999 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:52.439628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5759cb7889-qrq2v" event={"ID":"0e52e651-27a3-4c53-b525-066b26f3a67e","Type":"ContainerStarted","Data":"7dac25d67477eeab2da67787b7c4c44fe6d69dd93c9952d2daa5f07e7f8033b8"}
Jul 27 21:46:56.464729 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.464628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qm5vp" event={"ID":"71c0cb96-1bc0-411c-88bf-2137aae6a264","Type":"ContainerStarted","Data":"deeba737560ab2efaf9fdf77428edb3bdc32ebcb41e83a0b70efa09e17a911c8"}
Jul 27 21:46:56.468299 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.468265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"}
Jul 27 21:46:56.468441 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.468349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"}
Jul 27 21:46:56.468441 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.468393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"}
Jul 27 21:46:56.468441 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.468430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"}
Jul 27 21:46:56.468639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:56.468495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"}
Jul 27 21:46:57.477405 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:57.477324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerStarted","Data":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"}
Jul 27 21:46:57.505496 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:57.505417    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.822745622 podStartE2EDuration="13.505393677s" totalImagesPullingTime="6.654797708s" totalInitContainerRuntime="27.850347ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:57.503521409 +0000 UTC m=+83.156950052" watchObservedRunningTime="2026-07-27 21:46:57.505393677 +0000 UTC m=+83.158822322"
Jul 27 21:46:57.506022 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:57.505982    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5759cb7889-qrq2v" podStartSLOduration=6.505968178 podStartE2EDuration="6.505968178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:46:52.46068971 +0000 UTC m=+78.114118353" watchObservedRunningTime="2026-07-27 21:46:57.505968178 +0000 UTC m=+83.159396824"
Jul 27 21:46:58.450697 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:58.450642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6ddd7657b4-qcvcq"
Jul 27 21:46:59.430724 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:46:59.430687    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:47:00.507179 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:00.507141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qm5vp" event={"ID":"71c0cb96-1bc0-411c-88bf-2137aae6a264","Type":"ContainerStarted","Data":"fc0df1d0f4b59472523b50f15e8b771d97717e06c5776bf3e18135a6f145d77d"}
Jul 27 21:47:01.628929 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:01.628893    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:47:01.629394 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:01.628962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:47:01.634145 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:01.634121    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:47:02.416969 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:02.416925    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:47:02.417153 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:02.417038    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:47:02.521833 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:02.521805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:47:02.575468 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:02.575418    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:47:03.523945 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:03.523914    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qm5vp" event={"ID":"71c0cb96-1bc0-411c-88bf-2137aae6a264","Type":"ContainerStarted","Data":"5c495651e970b192d34f6ec061d980d42a7194dff7042b307d3f7de5e624b0af"}
Jul 27 21:47:03.544151 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:03.544097    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qm5vp" podStartSLOduration=20.44777199 podStartE2EDuration="34.544079381s" totalImagesPullingTime="14.096307391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 21:46:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:47:03.543126987 +0000 UTC m=+89.196555643" watchObservedRunningTime="2026-07-27 21:47:03.544079381 +0000 UTC m=+89.197508024"
Jul 27 21:47:08.473691 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.473647    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cf788c7df-hflhf" podUID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" containerName="console" containerID="cri-o://c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912" gracePeriod=15
Jul 27 21:47:08.731594 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.731534    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:47:08.841155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841117    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841352 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841167    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841352 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841189    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841352 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841210    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841352 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841254    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841352 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgk7b\" (UniqueName: \"kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b\") pod \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\" (UID: \"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55\") "
Jul 27 21:47:08.841702 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841676    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config" (OuterVolumeSpecName: "console-config") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:08.841702 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:08.841816 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.841688    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca" (OuterVolumeSpecName: "service-ca") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:08.843872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.843827    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:08.843974 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.843868    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:08.843974 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.843869    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b" (OuterVolumeSpecName: "kube-api-access-hgk7b") pod "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" (UID: "d3df6ff7-95a5-401c-9919-f7b2b7e0fe55"). InnerVolumeSpecName "kube-api-access-hgk7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:08.942888 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942855    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:08.942888 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942882    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:08.942888 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942891    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-oauth-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:08.943104 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942902    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:08.943104 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942910    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgk7b\" (UniqueName: \"kubernetes.io/projected/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-kube-api-access-hgk7b\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:08.943104 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:08.942918    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55-console-oauth-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:09.550961 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.550925    2560 generic.go:356] "Generic (PLEG): container finished" podID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" containerID="c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912" exitCode=0
Jul 27 21:47:09.551426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.550977    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cf788c7df-hflhf"
Jul 27 21:47:09.551426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.550993    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf788c7df-hflhf" event={"ID":"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55","Type":"ContainerDied","Data":"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"}
Jul 27 21:47:09.551426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.551040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cf788c7df-hflhf" event={"ID":"d3df6ff7-95a5-401c-9919-f7b2b7e0fe55","Type":"ContainerDied","Data":"539d083bf483d39e7259e3b9ae9e8cab272f2b2ed90921a4d7b08a09d9980b35"}
Jul 27 21:47:09.551426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.551068    2560 scope.go:117] "RemoveContainer" containerID="c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"
Jul 27 21:47:09.559707 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.559688    2560 scope.go:117] "RemoveContainer" containerID="c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"
Jul 27 21:47:09.559994 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:09.559972    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912\": container with ID starting with c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912 not found: ID does not exist" containerID="c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"
Jul 27 21:47:09.560055 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.560000    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912"} err="failed to get container status \"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912\": rpc error: code = NotFound desc = could not find container \"c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912\": container with ID starting with c806e415afbba7c3e14d8d362c9d4eb167099ab09bf5bbb51b29313a81d85912 not found: ID does not exist"
Jul 27 21:47:09.566749 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.566729    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:47:09.570025 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:09.570003    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cf788c7df-hflhf"]
Jul 27 21:47:10.915139 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:10.915108    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" path="/var/lib/kubelet/pods/d3df6ff7-95a5-401c-9919-f7b2b7e0fe55/volumes"
Jul 27 21:47:20.415098 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:20.415066    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qlsm7"
Jul 27 21:47:22.422324 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:22.422293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:47:22.426270 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:22.426242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7b669bc878-mxn88"
Jul 27 21:47:27.601989 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:27.601945    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bcf5f57c-pgs4r" podUID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" containerName="console" containerID="cri-o://cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039" gracePeriod=15
Jul 27 21:47:28.400776 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.400716    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524795    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqfkx\" (UniqueName: \"kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524845    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524869    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524889    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524930    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524945    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.524997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config\") pod \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\" (UID: \"662f9e21-0c25-451e-b6ff-d63dba87bd0e\") "
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.525310    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config" (OuterVolumeSpecName: "console-config") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.525404    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca" (OuterVolumeSpecName: "service-ca") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.525410    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:28.527751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.525415    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:28.528317 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.528130    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx" (OuterVolumeSpecName: "kube-api-access-xqfkx") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "kube-api-access-xqfkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:28.528317 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.528139    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:28.528317 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.528216    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "662f9e21-0c25-451e-b6ff-d63dba87bd0e" (UID: "662f9e21-0c25-451e-b6ff-d63dba87bd0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:28.625931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625888    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfkx\" (UniqueName: \"kubernetes.io/projected/662f9e21-0c25-451e-b6ff-d63dba87bd0e-kube-api-access-xqfkx\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.625931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625917    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.625931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625927    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.625931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625935    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-oauth-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.625931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625943    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.626397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625951    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/662f9e21-0c25-451e-b6ff-d63dba87bd0e-trusted-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.626397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.625958    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/662f9e21-0c25-451e-b6ff-d63dba87bd0e-console-oauth-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:28.632795 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.632765    2560 generic.go:356] "Generic (PLEG): container finished" podID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" containerID="cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039" exitCode=0
Jul 27 21:47:28.632938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.632818    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bcf5f57c-pgs4r"
Jul 27 21:47:28.632938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.632853    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf5f57c-pgs4r" event={"ID":"662f9e21-0c25-451e-b6ff-d63dba87bd0e","Type":"ContainerDied","Data":"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"}
Jul 27 21:47:28.632938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.632905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bcf5f57c-pgs4r" event={"ID":"662f9e21-0c25-451e-b6ff-d63dba87bd0e","Type":"ContainerDied","Data":"2f96b2a9633d4ba5d1f1f76fa470271dd336fe03f4aa824f5fcc5aed24ca8264"}
Jul 27 21:47:28.632938 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.632925    2560 scope.go:117] "RemoveContainer" containerID="cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"
Jul 27 21:47:28.641696 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.641681    2560 scope.go:117] "RemoveContainer" containerID="cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"
Jul 27 21:47:28.641969 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:28.641952    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039\": container with ID starting with cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039 not found: ID does not exist" containerID="cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"
Jul 27 21:47:28.642023 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.641976    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039"} err="failed to get container status \"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039\": rpc error: code = NotFound desc = could not find container \"cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039\": container with ID starting with cfead37131e80381e0111eb4b855d6a99c39cfd8186afcc4bf3400e704dca039 not found: ID does not exist"
Jul 27 21:47:28.652610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.652581    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:47:28.657177 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.657157    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bcf5f57c-pgs4r"]
Jul 27 21:47:28.914585 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:28.914555    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" path="/var/lib/kubelet/pods/662f9e21-0c25-451e-b6ff-d63dba87bd0e/volumes"
Jul 27 21:47:44.430348 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:44.430315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:47:44.450503 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:44.450476    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:47:44.713834 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:44.713761    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:47:58.088592 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.088556    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.088925    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" containerName="console"
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.088938    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" containerName="console"
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.088951    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" containerName="console"
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.088957    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" containerName="console"
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.089004    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3df6ff7-95a5-401c-9919-f7b2b7e0fe55" containerName="console"
Jul 27 21:47:58.089057 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.089013    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="662f9e21-0c25-451e-b6ff-d63dba87bd0e" containerName="console"
Jul 27 21:47:58.092490 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.092473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.101668 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.101640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:47:58.182717 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182677    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182717 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hsg\" (UniqueName: \"kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182749    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.182928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.182894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.261986 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.261952    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:47:58.262327 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262286    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="alertmanager" containerID="cri-o://b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9" gracePeriod=120
Jul 27 21:47:58.262480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262350    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-metric" containerID="cri-o://fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb" gracePeriod=120
Jul 27 21:47:58.262480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262421    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy" containerID="cri-o://c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5" gracePeriod=120
Jul 27 21:47:58.262480 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262430    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="config-reloader" containerID="cri-o://f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209" gracePeriod=120
Jul 27 21:47:58.262686 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262378    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-web" containerID="cri-o://95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00" gracePeriod=120
Jul 27 21:47:58.262686 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.262379    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="prom-label-proxy" containerID="cri-o://735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb" gracePeriod=120
Jul 27 21:47:58.283835 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.283942 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.283942 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.283942 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hsg\" (UniqueName: \"kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.283942 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.284132 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.284132 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.283998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.284830 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.284770    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.284830 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.284796    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.284830 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.284802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.285338 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.285313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.286431 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.286354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.286620 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.286601    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.292985 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.292965    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hsg\" (UniqueName: \"kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg\") pod \"console-55b55cf6b6-sgxrj\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") " pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.403032 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.402986    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:47:58.544293 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.544239    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:47:58.546537 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:47:58.546512    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1da729cd_43d0_4df3_bf1a_d8b7a7355742.slice/crio-60726d6f8c4c4ef99921b5deed72d232e2826e14c8bc3337b8ffe660dae39701 WatchSource:0}: Error finding container 60726d6f8c4c4ef99921b5deed72d232e2826e14c8bc3337b8ffe660dae39701: Status 404 returned error can't find the container with id 60726d6f8c4c4ef99921b5deed72d232e2826e14c8bc3337b8ffe660dae39701
Jul 27 21:47:58.756268 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.756159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b55cf6b6-sgxrj" event={"ID":"1da729cd-43d0-4df3-bf1a-d8b7a7355742","Type":"ContainerStarted","Data":"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"}
Jul 27 21:47:58.756268 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.756231    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b55cf6b6-sgxrj" event={"ID":"1da729cd-43d0-4df3-bf1a-d8b7a7355742","Type":"ContainerStarted","Data":"60726d6f8c4c4ef99921b5deed72d232e2826e14c8bc3337b8ffe660dae39701"}
Jul 27 21:47:58.759521 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759496    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb" exitCode=0
Jul 27 21:47:58.759521 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759518    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5" exitCode=0
Jul 27 21:47:58.759521 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759525    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209" exitCode=0
Jul 27 21:47:58.759755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759532    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9" exitCode=0
Jul 27 21:47:58.759755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"}
Jul 27 21:47:58.759755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"}
Jul 27 21:47:58.759755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"}
Jul 27 21:47:58.759755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.759685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"}
Jul 27 21:47:58.776861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:58.776807    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55b55cf6b6-sgxrj" podStartSLOduration=0.776792042 podStartE2EDuration="776.792042ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:47:58.77487442 +0000 UTC m=+144.428303075" watchObservedRunningTime="2026-07-27 21:47:58.776792042 +0000 UTC m=+144.430220684"
Jul 27 21:47:59.520377 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.520353    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.595855 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595763    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.595855 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595809    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.595855 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595883    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595910    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.595951    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596316 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.596144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596316 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.596222    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:59.596764 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.596181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.596972 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.596953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.597300 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.597283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.597479 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.597441    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.597632 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.597617    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4sj4\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4\") pod \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\" (UID: \"509f17ca-db52-4b93-bd5e-eb81f3cbb570\") "
Jul 27 21:47:59.597812 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.597032    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:47:59.598087 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.598059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:47:59.598287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.598253    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-main-db\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.598385 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.598373    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.600633 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.600596    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume" (OuterVolumeSpecName: "config-volume") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.600854 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.600825    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.600931 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.600829    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:59.601018 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.600998    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.601601 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.601578    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out" (OuterVolumeSpecName: "config-out") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:47:59.601698 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.601599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4" (OuterVolumeSpecName: "kube-api-access-g4sj4") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "kube-api-access-g4sj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:47:59.601933 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.601905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.602046 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.602017    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.604561 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.604536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.612279 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.612253    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config" (OuterVolumeSpecName: "web-config") pod "509f17ca-db52-4b93-bd5e-eb81f3cbb570" (UID: "509f17ca-db52-4b93-bd5e-eb81f3cbb570"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:47:59.698978 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.698938    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-cluster-tls-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.698978 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.698978    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.698995    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-tls-assets\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699008    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-main-tls\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699021    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-web-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699034    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699046    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699060    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509f17ca-db52-4b93-bd5e-eb81f3cbb570-metrics-client-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699075    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/509f17ca-db52-4b93-bd5e-eb81f3cbb570-config-out\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699088    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/509f17ca-db52-4b93-bd5e-eb81f3cbb570-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.699202 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.699100    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4sj4\" (UniqueName: \"kubernetes.io/projected/509f17ca-db52-4b93-bd5e-eb81f3cbb570-kube-api-access-g4sj4\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:47:59.766751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766713    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb" exitCode=0
Jul 27 21:47:59.766751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766742    2560 generic.go:356] "Generic (PLEG): container finished" podID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerID="95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00" exitCode=0
Jul 27 21:47:59.767004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766799    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.767004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"}
Jul 27 21:47:59.767004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"}
Jul 27 21:47:59.767004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"509f17ca-db52-4b93-bd5e-eb81f3cbb570","Type":"ContainerDied","Data":"5530b1f09cf6401ddfc5936367cfeea15815c06f22174c2716cc7bc42c089e8c"}
Jul 27 21:47:59.767004 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.766961    2560 scope.go:117] "RemoveContainer" containerID="735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"
Jul 27 21:47:59.776099 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.776082    2560 scope.go:117] "RemoveContainer" containerID="fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"
Jul 27 21:47:59.783601 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.783583    2560 scope.go:117] "RemoveContainer" containerID="c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"
Jul 27 21:47:59.791203 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.791182    2560 scope.go:117] "RemoveContainer" containerID="95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"
Jul 27 21:47:59.793426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.793404    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:47:59.799545 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.799462    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:47:59.799644 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.799576    2560 scope.go:117] "RemoveContainer" containerID="f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"
Jul 27 21:47:59.809552 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.809529    2560 scope.go:117] "RemoveContainer" containerID="b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"
Jul 27 21:47:59.817460 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.817429    2560 scope.go:117] "RemoveContainer" containerID="9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"
Jul 27 21:47:59.825585 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.825561    2560 scope.go:117] "RemoveContainer" containerID="735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"
Jul 27 21:47:59.825841 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.825821    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb\": container with ID starting with 735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb not found: ID does not exist" containerID="735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"
Jul 27 21:47:59.825890 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.825851    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"} err="failed to get container status \"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb\": rpc error: code = NotFound desc = could not find container \"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb\": container with ID starting with 735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb not found: ID does not exist"
Jul 27 21:47:59.825890 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.825870    2560 scope.go:117] "RemoveContainer" containerID="fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"
Jul 27 21:47:59.826132 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.826118    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb\": container with ID starting with fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb not found: ID does not exist" containerID="fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"
Jul 27 21:47:59.826170 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826146    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"} err="failed to get container status \"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb\": rpc error: code = NotFound desc = could not find container \"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb\": container with ID starting with fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb not found: ID does not exist"
Jul 27 21:47:59.826170 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826160    2560 scope.go:117] "RemoveContainer" containerID="c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"
Jul 27 21:47:59.826393 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.826377    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5\": container with ID starting with c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5 not found: ID does not exist" containerID="c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"
Jul 27 21:47:59.826427 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826402    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"} err="failed to get container status \"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5\": rpc error: code = NotFound desc = could not find container \"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5\": container with ID starting with c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5 not found: ID does not exist"
Jul 27 21:47:59.826427 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826420    2560 scope.go:117] "RemoveContainer" containerID="95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"
Jul 27 21:47:59.826676 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.826659    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00\": container with ID starting with 95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00 not found: ID does not exist" containerID="95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"
Jul 27 21:47:59.826716 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826681    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"} err="failed to get container status \"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00\": rpc error: code = NotFound desc = could not find container \"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00\": container with ID starting with 95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00 not found: ID does not exist"
Jul 27 21:47:59.826716 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826695    2560 scope.go:117] "RemoveContainer" containerID="f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"
Jul 27 21:47:59.826932 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.826917    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209\": container with ID starting with f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209 not found: ID does not exist" containerID="f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"
Jul 27 21:47:59.826988 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826934    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"} err="failed to get container status \"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209\": rpc error: code = NotFound desc = could not find container \"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209\": container with ID starting with f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209 not found: ID does not exist"
Jul 27 21:47:59.826988 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.826951    2560 scope.go:117] "RemoveContainer" containerID="b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"
Jul 27 21:47:59.827153 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.827136    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9\": container with ID starting with b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9 not found: ID does not exist" containerID="b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"
Jul 27 21:47:59.827191 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827157    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"} err="failed to get container status \"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9\": rpc error: code = NotFound desc = could not find container \"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9\": container with ID starting with b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9 not found: ID does not exist"
Jul 27 21:47:59.827191 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827172    2560 scope.go:117] "RemoveContainer" containerID="9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"
Jul 27 21:47:59.827387 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:47:59.827370    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24\": container with ID starting with 9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24 not found: ID does not exist" containerID="9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"
Jul 27 21:47:59.827442 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827389    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"} err="failed to get container status \"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24\": rpc error: code = NotFound desc = could not find container \"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24\": container with ID starting with 9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24 not found: ID does not exist"
Jul 27 21:47:59.827442 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827401    2560 scope.go:117] "RemoveContainer" containerID="735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"
Jul 27 21:47:59.827647 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827632    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb"} err="failed to get container status \"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb\": rpc error: code = NotFound desc = could not find container \"735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb\": container with ID starting with 735e762e1de8ee6b9defebd9db07b732527733cd83e6b6f428c7175de84a32eb not found: ID does not exist"
Jul 27 21:47:59.827647 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827647    2560 scope.go:117] "RemoveContainer" containerID="fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"
Jul 27 21:47:59.827844 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827829    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb"} err="failed to get container status \"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb\": rpc error: code = NotFound desc = could not find container \"fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb\": container with ID starting with fb109db55691bdf43ac9d8afeec5695e28725c78c22be11331853f43c31ec0eb not found: ID does not exist"
Jul 27 21:47:59.827844 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.827844    2560 scope.go:117] "RemoveContainer" containerID="c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"
Jul 27 21:47:59.828055 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828039    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5"} err="failed to get container status \"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5\": rpc error: code = NotFound desc = could not find container \"c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5\": container with ID starting with c8d48eb533a85d1cb9beb9b285f621ed81c12a7f3c3890d6fe2a15044dc7f7f5 not found: ID does not exist"
Jul 27 21:47:59.828101 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828055    2560 scope.go:117] "RemoveContainer" containerID="95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"
Jul 27 21:47:59.828260 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828243    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00"} err="failed to get container status \"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00\": rpc error: code = NotFound desc = could not find container \"95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00\": container with ID starting with 95d172aa7065fbd81dd091d30d485e8867cf8e8c37baa20fccf7d591b3a6fd00 not found: ID does not exist"
Jul 27 21:47:59.828301 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828261    2560 scope.go:117] "RemoveContainer" containerID="f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"
Jul 27 21:47:59.828479 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828444    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209"} err="failed to get container status \"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209\": rpc error: code = NotFound desc = could not find container \"f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209\": container with ID starting with f1b82f4bd4525f6ed1438754facdb4acd95ed5363443f0071cb24e3951d30209 not found: ID does not exist"
Jul 27 21:47:59.828479 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828479    2560 scope.go:117] "RemoveContainer" containerID="b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"
Jul 27 21:47:59.828680 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828665    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9"} err="failed to get container status \"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9\": rpc error: code = NotFound desc = could not find container \"b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9\": container with ID starting with b03ceac0b4abe9c86c47ed6c4df073538fbbd2003b46d0057a0c9de52a09cfb9 not found: ID does not exist"
Jul 27 21:47:59.828717 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828680    2560 scope.go:117] "RemoveContainer" containerID="9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"
Jul 27 21:47:59.828865 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.828851    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24"} err="failed to get container status \"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24\": rpc error: code = NotFound desc = could not find container \"9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24\": container with ID starting with 9362e27da770a94b61e4141921f1e055f5fd97514a59277397a82cd2e8ac1f24 not found: ID does not exist"
Jul 27 21:47:59.833045 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833024    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:47:59.833332 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833321    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-metric"
Jul 27 21:47:59.833369 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833334    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-metric"
Jul 27 21:47:59.833369 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833345    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-web"
Jul 27 21:47:59.833369 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833350    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-web"
Jul 27 21:47:59.833369 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833367    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="alertmanager"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833372    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="alertmanager"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833385    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="init-config-reloader"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833390    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="init-config-reloader"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833395    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="config-reloader"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833401    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="config-reloader"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833409    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="prom-label-proxy"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833413    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="prom-label-proxy"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833420    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833425    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833489    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="alertmanager"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833497    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-metric"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy"
Jul 27 21:47:59.833509 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833510    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="prom-label-proxy"
Jul 27 21:47:59.833856 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833516    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="config-reloader"
Jul 27 21:47:59.833856 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.833522    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" containerName="kube-rbac-proxy-web"
Jul 27 21:47:59.838764 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.838749    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.843287 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kgq96\""
Jul 27 21:47:59.843391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 27 21:47:59.843391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843272    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 27 21:47:59.843592 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843579    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 27 21:47:59.843786 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843772    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 27 21:47:59.843823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.843808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 27 21:47:59.844086 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.844072    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 27 21:47:59.844227 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.844216    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 27 21:47:59.844322 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.844307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 27 21:47:59.851910 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.851491    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 27 21:47:59.859231 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.859208    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:47:59.900285 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900285 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900544 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-out\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900544 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900544 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900527    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900801 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900577    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900801 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900693    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-web-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900801 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s479\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-kube-api-access-5s479\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900959 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900959 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900959 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900896    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.900959 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:47:59.901126 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:47:59.900974    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.001937 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.001889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-web-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.001937 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.001939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s479\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-kube-api-access-5s479\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002060    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002253    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-out\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.002569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.002526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.003365 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.003336    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.003858 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.003664    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.004569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.004192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.005439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.005412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-web-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.005657 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.005609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-volume\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.005785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.005758    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.005785 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.005769    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.006215 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.006192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.006414 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.006395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-config-out\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.006504 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.006419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.007019 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.006998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.007130 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.007112    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.011867 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.011846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s479\" (UniqueName: \"kubernetes.io/projected/d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4-kube-api-access-5s479\") pod \"alertmanager-main-0\" (UID: \"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.149425 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.149334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 27 21:48:00.296571 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.296542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 27 21:48:00.298704 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:48:00.298674    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83fa88e_3a53_4fb8_8d48_1e4bc1a60ad4.slice/crio-be273aa9381f306483e0521780f6ef8676a06b56a724724930df01bb2ba3c162 WatchSource:0}: Error finding container be273aa9381f306483e0521780f6ef8676a06b56a724724930df01bb2ba3c162: Status 404 returned error can't find the container with id be273aa9381f306483e0521780f6ef8676a06b56a724724930df01bb2ba3c162
Jul 27 21:48:00.772892 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.772855    2560 generic.go:356] "Generic (PLEG): container finished" podID="d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4" containerID="77d83fd988baf0999187dbd22c264d5b5ff135220f9502456495738aee5807a3" exitCode=0
Jul 27 21:48:00.773307 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.772947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerDied","Data":"77d83fd988baf0999187dbd22c264d5b5ff135220f9502456495738aee5807a3"}
Jul 27 21:48:00.773307 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.773030    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"be273aa9381f306483e0521780f6ef8676a06b56a724724930df01bb2ba3c162"}
Jul 27 21:48:00.919033 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:00.919005    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509f17ca-db52-4b93-bd5e-eb81f3cbb570" path="/var/lib/kubelet/pods/509f17ca-db52-4b93-bd5e-eb81f3cbb570/volumes"
Jul 27 21:48:01.782058 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"568e3941c556462e3036c585a7d34cdada60dafad9973679845180b1038de1af"}
Jul 27 21:48:01.782429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782087    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"b8b4eb6d8dd3d1915331fa2a9bf46f3ab840ff8d5a7962335650d45a4f7ba79b"}
Jul 27 21:48:01.782429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782112    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"fe5d2a3ca021ff642bbffecee99481efda7ae0cccdc442469ae20b73b4da0714"}
Jul 27 21:48:01.782429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782143    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"831cf77824b6f86eb76ce5a75514f082ba966f532e61b6c8707a1ee9f7a72eea"}
Jul 27 21:48:01.782429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"e67459dacd8043580278dfe5a7469796fc604c8fe1bebfc69a61a6dedb0fbf02"}
Jul 27 21:48:01.782429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.782191    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4","Type":"ContainerStarted","Data":"1e3832201f34c63a1b46610d329cc6f4b58f1c0da59a70d1b694638077deb31f"}
Jul 27 21:48:01.812797 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:01.812732    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.796055983 podStartE2EDuration="2.812714103s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.65812ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:48:01.811211259 +0000 UTC m=+147.464639923" watchObservedRunningTime="2026-07-27 21:48:01.812714103 +0000 UTC m=+147.466143097"
Jul 27 21:48:02.488876 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.488842    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:02.489164 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489139    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="prometheus" containerID="cri-o://70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb" gracePeriod=600
Jul 27 21:48:02.489239 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489200    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3" gracePeriod=600
Jul 27 21:48:02.489398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489240    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="thanos-sidecar" containerID="cri-o://770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c" gracePeriod=600
Jul 27 21:48:02.489398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489251    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy" containerID="cri-o://d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787" gracePeriod=600
Jul 27 21:48:02.489398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489280    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="config-reloader" containerID="cri-o://51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85" gracePeriod=600
Jul 27 21:48:02.489398 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.489230    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-web" containerID="cri-o://32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84" gracePeriod=600
Jul 27 21:48:02.743791 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.743715    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:02.797391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797356    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3" exitCode=0
Jul 27 21:48:02.797391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797389    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787" exitCode=0
Jul 27 21:48:02.797391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797398    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84" exitCode=0
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797409    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c" exitCode=0
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797419    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85" exitCode=0
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797427    2560 generic.go:356] "Generic (PLEG): container finished" podID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb" exitCode=0
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797495    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797644    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797683    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797689    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797733    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797776    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"}
Jul 27 21:48:02.797952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.797816    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a83cd7e2-8359-44b4-916c-911fc769cf7d","Type":"ContainerDied","Data":"7839f414aba65ef71c585afd8af92eae7733cdccb0fcae07ce2a6ebc96210322"}
Jul 27 21:48:02.807042 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.807021    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.818484 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.818439    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.827745 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827716    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.827881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827768    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.827881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827798    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.827881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827825    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.827881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827853    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827910    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827969    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwc42\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.827997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828066    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828155 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828159    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828205    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828240    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828307    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828331    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle\") pod \"a83cd7e2-8359-44b4-916c-911fc769cf7d\" (UID: \"a83cd7e2-8359-44b4-916c-911fc769cf7d\") "
Jul 27 21:48:02.828599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828440    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.828928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828661    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.828928 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.828848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.829288 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.829094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.829809 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.829783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.829895 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.829788    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.830495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.830258    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:48:02.830495 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.830272    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.830794 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.830661    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.833091 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.833054    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config" (OuterVolumeSpecName: "config") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.833954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.833847    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:02.833954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.833860    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.834200 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834178    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:48:02.834335 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834291    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42" (OuterVolumeSpecName: "kube-api-access-xwc42") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "kube-api-access-xwc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:48:02.834474 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.834539 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.834594 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.834678 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.834661    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out" (OuterVolumeSpecName: "config-out") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:48:02.835415 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.835384    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.835553 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.835530    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.836421 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.836397    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.841093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.841075    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.848864 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.848822    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config" (OuterVolumeSpecName: "web-config") pod "a83cd7e2-8359-44b4-916c-911fc769cf7d" (UID: "a83cd7e2-8359-44b4-916c-911fc769cf7d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:02.848927 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.848889    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.856443 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.856425    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.863687 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.863669    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.863971 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.863952    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.864020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.863981    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.864020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864003    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.864262 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.864245    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.864302 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864269    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.864302 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864286    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.864572 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.864552    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.864622 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864580    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.864622 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864602    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.864855 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.864836    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.864932 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864866    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.864932 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.864888    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.865140 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.865121    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.865196 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865145    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.865196 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865160    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.865398 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.865378    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.865524 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865403    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.865524 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865417    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.865657 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:02.865638    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.865708 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865661    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.865708 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865674    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.865895 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865866    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.865966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.865897    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.866178 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866157    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.866254 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866179    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.866482 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866440    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.866569 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866483    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.866751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866728    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.866799 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866752    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.866985 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866966    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.867038 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.866986    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.867217 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867192    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.867296 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867219    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.867511 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867488    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.867511 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867509    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.867787 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867765    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.867787 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.867786    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.868063 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868037    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.868132 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868067    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.868312 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868291    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.868358 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868316    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.868612 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868588    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.868671 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868615    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.868866 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868848    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.868917 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.868866    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.869093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869075    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.869093 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869094    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.869356 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869339    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.869406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869357    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.869597 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869581    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.869655 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869598    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.869780 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869760    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.869841 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.869783    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.870029 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870012    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.870090 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870030    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.870294 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870276    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.870393 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870297    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.870573 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870545    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.870573 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870564    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.870767 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870749    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.870859 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.870769    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.871047 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871030    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.871110 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871048    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.871242 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871225    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.871285 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871242    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.871537 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871511    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.871599 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871540    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.871771 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871752    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.871771 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.871773    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.872032 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872006    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.872032 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872028    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.872264 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872245    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.872328 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872264    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.872530 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872512    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.872530 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872530    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.872758 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872733    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.872806 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872758    2560 scope.go:117] "RemoveContainer" containerID="f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"
Jul 27 21:48:02.872952 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872934    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3"} err="failed to get container status \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": rpc error: code = NotFound desc = could not find container \"f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3\": container with ID starting with f2e0c279563dca9d75ecf883d63dcfc219745852b7e8217c2ee8c7fa568901f3 not found: ID does not exist"
Jul 27 21:48:02.872994 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.872953    2560 scope.go:117] "RemoveContainer" containerID="d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"
Jul 27 21:48:02.873200 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873180    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787"} err="failed to get container status \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": rpc error: code = NotFound desc = could not find container \"d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787\": container with ID starting with d1f79979f3e20c9dc92ee47be39c5fa27bb6aecf50d478e026242fd199472787 not found: ID does not exist"
Jul 27 21:48:02.873246 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873202    2560 scope.go:117] "RemoveContainer" containerID="32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"
Jul 27 21:48:02.873463 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873431    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84"} err="failed to get container status \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": rpc error: code = NotFound desc = could not find container \"32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84\": container with ID starting with 32f0d9bfcf0d4a5359caebd5c94d0ab5667e013d14e9226dd38d6142a3f7ac84 not found: ID does not exist"
Jul 27 21:48:02.873519 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873466    2560 scope.go:117] "RemoveContainer" containerID="770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"
Jul 27 21:48:02.873713 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873693    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c"} err="failed to get container status \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": rpc error: code = NotFound desc = could not find container \"770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c\": container with ID starting with 770ab7a7f00bddf7b06fdad19febb86ddcbc55facc2186fd6fe75939fc578c7c not found: ID does not exist"
Jul 27 21:48:02.873751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873715    2560 scope.go:117] "RemoveContainer" containerID="51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"
Jul 27 21:48:02.873918 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873902    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85"} err="failed to get container status \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": rpc error: code = NotFound desc = could not find container \"51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85\": container with ID starting with 51521e262e0a41782730783970dfc7077d589f3bd73924243f17e7fffb4dea85 not found: ID does not exist"
Jul 27 21:48:02.873955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.873918    2560 scope.go:117] "RemoveContainer" containerID="70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"
Jul 27 21:48:02.874126 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.874105    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb"} err="failed to get container status \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": rpc error: code = NotFound desc = could not find container \"70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb\": container with ID starting with 70f68ac9ff6b23196010f2d987e272e26cdc20220fb1050d15a0602cc67958fb not found: ID does not exist"
Jul 27 21:48:02.874177 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.874129    2560 scope.go:117] "RemoveContainer" containerID="75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"
Jul 27 21:48:02.874365 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.874348    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c"} err="failed to get container status \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": rpc error: code = NotFound desc = could not find container \"75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c\": container with ID starting with 75a8bd67b2ecd6ff0e6eb7ee67ae9fbaadcad81c39abc00618b55abd547b079c not found: ID does not exist"
Jul 27 21:48:02.929820 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929792    2560 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-metrics-client-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.929820 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929822    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929839    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-tls-assets\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929855    2560 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929870    2560 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-grpc-tls\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929884    2560 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-kube-rbac-proxy\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929899    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-config-out\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929920    2560 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929934    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929960    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwc42\" (UniqueName: \"kubernetes.io/projected/a83cd7e2-8359-44b4-916c-911fc769cf7d-kube-api-access-xwc42\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929975    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929983    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-metrics-client-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.929998    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-db\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930011    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930019    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-web-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930027    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930035    2560 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930043    2560 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a83cd7e2-8359-44b4-916c-911fc769cf7d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:02.930137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:02.930055    2560 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a83cd7e2-8359-44b4-916c-911fc769cf7d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:03.118788 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.118753    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:03.128394 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.128365    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:03.162872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.162841    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:03.163171 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163158    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="thanos-sidecar"
Jul 27 21:48:03.163212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163173    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="thanos-sidecar"
Jul 27 21:48:03.163212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163188    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="prometheus"
Jul 27 21:48:03.163212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163194    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="prometheus"
Jul 27 21:48:03.163212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163201    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="config-reloader"
Jul 27 21:48:03.163212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163206    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="config-reloader"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163220    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="init-config-reloader"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163226    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="init-config-reloader"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163233    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-web"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163238    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-web"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163248    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-thanos"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163255    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-thanos"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163262    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163267    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163325    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="thanos-sidecar"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163334    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="config-reloader"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163339    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-web"
Jul 27 21:48:03.163346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163345    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy-thanos"
Jul 27 21:48:03.163683 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163352    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="prometheus"
Jul 27 21:48:03.163683 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.163357    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" containerName="kube-rbac-proxy"
Jul 27 21:48:03.169174 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.169152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.172024 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.171989    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 27 21:48:03.172264 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.172237    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 27 21:48:03.172338 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.172235    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 27 21:48:03.173020 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173003    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-975fml6qnmh2g\""
Jul 27 21:48:03.173357 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173335    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 27 21:48:03.173478 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173403    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 27 21:48:03.173634 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173616    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 27 21:48:03.173634 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173620    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 27 21:48:03.173775 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173651    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 27 21:48:03.173775 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 27 21:48:03.173981 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173965    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 27 21:48:03.174035 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.173967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 27 21:48:03.174351 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.174336    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 27 21:48:03.175262 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.175222    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gjnbj\""
Jul 27 21:48:03.175668 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.175599    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 27 21:48:03.179824 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.179802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 27 21:48:03.190118 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.190087    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:03.232363 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232363 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232610 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232659    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-web-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232731    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232768    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxzg\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-kube-api-access-hbxzg\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.232872 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232858    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.233120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.233120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.233120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232922    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-config-out\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.233120 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.232966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333732 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333693    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxzg\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-kube-api-access-hbxzg\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333732 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333967 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333751    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333967 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333967 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333807    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-config-out\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.333967 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.333990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334144 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334163    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-web-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334395 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334823 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.334935 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.334840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.335609 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.335583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337106 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337080    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-config-out\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337236 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337202    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337568 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337909 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337716    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337909 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.337909 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.338096 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.337955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.338213 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.338191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.338337 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.338315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.340000 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.339974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/463d472c-cc06-4e60-8924-8e757d17991a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.340135 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.340113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.340216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.340201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.340279 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.340235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.340367 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.340351    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/463d472c-cc06-4e60-8924-8e757d17991a-web-config\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.343069 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.343051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxzg\" (UniqueName: \"kubernetes.io/projected/463d472c-cc06-4e60-8924-8e757d17991a-kube-api-access-hbxzg\") pod \"prometheus-k8s-0\" (UID: \"463d472c-cc06-4e60-8924-8e757d17991a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.479962 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.479864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:03.629714 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.629682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 27 21:48:03.633420 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:48:03.633388    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463d472c_cc06_4e60_8924_8e757d17991a.slice/crio-c35bd1db159b58782a2f98de037827c719ae04add58db4aa667f4f254bf75c3f WatchSource:0}: Error finding container c35bd1db159b58782a2f98de037827c719ae04add58db4aa667f4f254bf75c3f: Status 404 returned error can't find the container with id c35bd1db159b58782a2f98de037827c719ae04add58db4aa667f4f254bf75c3f
Jul 27 21:48:03.803608 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.803512    2560 generic.go:356] "Generic (PLEG): container finished" podID="463d472c-cc06-4e60-8924-8e757d17991a" containerID="cb3297a05f87c0dbf728ad385a97c9aa93b581ad480a31c08418c5a2d28a7c8a" exitCode=0
Jul 27 21:48:03.803995 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.803615    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerDied","Data":"cb3297a05f87c0dbf728ad385a97c9aa93b581ad480a31c08418c5a2d28a7c8a"}
Jul 27 21:48:03.803995 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:03.803727    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"c35bd1db159b58782a2f98de037827c719ae04add58db4aa667f4f254bf75c3f"}
Jul 27 21:48:04.811956 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.811905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"1b028be5a04f360311dd3166f896e3e68baabad6978332af00ab486332b02400"}
Jul 27 21:48:04.812397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.811989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"167ba754ad3cbb66f1ca7f4f8442d1df97d8a8dfbfe8923677a36edb5bb5b4f2"}
Jul 27 21:48:04.812397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.812028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"95345d28dcc2c5da1b844c1dacfd397eb1ac7030fae945f37e297491d0f8751d"}
Jul 27 21:48:04.812397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.812073    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"3d621b8d28f3596e6f7cc30c726cc90100df8be2b859678c9154e505831a413d"}
Jul 27 21:48:04.812397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.812113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"91f57e249da0accf410cc3634d1e9c581a9c2a67e52958e355cf9596244bd1ca"}
Jul 27 21:48:04.812397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.812151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"463d472c-cc06-4e60-8924-8e757d17991a","Type":"ContainerStarted","Data":"2a9fcfc8835afdc183a1ec1d34677a9d2a9116a7e5eb829de5b6278aff687036"}
Jul 27 21:48:04.848861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.848792    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.8297110760000002 podStartE2EDuration="1.848772186s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.06111ms" isStatefulPod=false podCreationTimestamp="2026-07-27 21:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:48:04.846603697 +0000 UTC m=+150.500032339" watchObservedRunningTime="2026-07-27 21:48:04.848772186 +0000 UTC m=+150.502200832"
Jul 27 21:48:04.915102 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:04.915060    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a83cd7e2-8359-44b4-916c-911fc769cf7d" path="/var/lib/kubelet/pods/a83cd7e2-8359-44b4-916c-911fc769cf7d/volumes"
Jul 27 21:48:08.403828 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.403786    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:48:08.404304 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.403854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:48:08.408792 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.408766    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:48:08.480316 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.480274    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:48:08.833293 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.833267    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:48:08.878835 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:08.878804    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:48:33.898746 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:33.898654    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5759cb7889-qrq2v" podUID="0e52e651-27a3-4c53-b525-066b26f3a67e" containerName="console" containerID="cri-o://a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b" gracePeriod=15
Jul 27 21:48:34.171769 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.171742    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:48:34.208072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208077    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208127    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208230    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208406 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208283    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bt7vc\" (UniqueName: \"kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc\") pod \"0e52e651-27a3-4c53-b525-066b26f3a67e\" (UID: \"0e52e651-27a3-4c53-b525-066b26f3a67e\") "
Jul 27 21:48:34.208722 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:34.208722 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208560    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:34.209072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.208944    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config" (OuterVolumeSpecName: "console-config") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:34.209072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.209037    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:48:34.210761 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.210727    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:34.210863 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.210810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:48:34.211029 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.211010    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc" (OuterVolumeSpecName: "kube-api-access-bt7vc") pod "0e52e651-27a3-4c53-b525-066b26f3a67e" (UID: "0e52e651-27a3-4c53-b525-066b26f3a67e"). InnerVolumeSpecName "kube-api-access-bt7vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:48:34.308955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308916    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-oauth-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.308955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308945    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e52e651-27a3-4c53-b525-066b26f3a67e-console-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.308955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308956    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-console-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.308955 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308965    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt7vc\" (UniqueName: \"kubernetes.io/projected/0e52e651-27a3-4c53-b525-066b26f3a67e-kube-api-access-bt7vc\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.309216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308974    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-trusted-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.309216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308983    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-oauth-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.309216 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.308991    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e52e651-27a3-4c53-b525-066b26f3a67e-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:48:34.935639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.935609    2560 generic.go:356] "Generic (PLEG): container finished" podID="0e52e651-27a3-4c53-b525-066b26f3a67e" containerID="a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b" exitCode=0
Jul 27 21:48:34.935966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.935663    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5759cb7889-qrq2v"
Jul 27 21:48:34.935966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.935687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5759cb7889-qrq2v" event={"ID":"0e52e651-27a3-4c53-b525-066b26f3a67e","Type":"ContainerDied","Data":"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"}
Jul 27 21:48:34.935966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.935736    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5759cb7889-qrq2v" event={"ID":"0e52e651-27a3-4c53-b525-066b26f3a67e","Type":"ContainerDied","Data":"7dac25d67477eeab2da67787b7c4c44fe6d69dd93c9952d2daa5f07e7f8033b8"}
Jul 27 21:48:34.935966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.935755    2560 scope.go:117] "RemoveContainer" containerID="a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"
Jul 27 21:48:34.943963 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.943939    2560 scope.go:117] "RemoveContainer" containerID="a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"
Jul 27 21:48:34.944242 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:48:34.944225    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b\": container with ID starting with a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b not found: ID does not exist" containerID="a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"
Jul 27 21:48:34.944300 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.944252    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b"} err="failed to get container status \"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b\": rpc error: code = NotFound desc = could not find container \"a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b\": container with ID starting with a842e8f6182782cf6745e93e47fe5af26986e863d45f3ee555ad0bb4cce5358b not found: ID does not exist"
Jul 27 21:48:34.954813 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.954787    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:48:34.959606 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:34.959563    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5759cb7889-qrq2v"]
Jul 27 21:48:36.914811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:48:36.914778    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e52e651-27a3-4c53-b525-066b26f3a67e" path="/var/lib/kubelet/pods/0e52e651-27a3-4c53-b525-066b26f3a67e/volumes"
Jul 27 21:49:03.480975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:03.480919    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:49:03.497388 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:03.497363    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:49:04.080609 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:04.080566    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 27 21:49:16.742082 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.742044    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z755f"]
Jul 27 21:49:16.742531 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.742514    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e52e651-27a3-4c53-b525-066b26f3a67e" containerName="console"
Jul 27 21:49:16.742580 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.742532    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e52e651-27a3-4c53-b525-066b26f3a67e" containerName="console"
Jul 27 21:49:16.742617 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.742605    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e52e651-27a3-4c53-b525-066b26f3a67e" containerName="console"
Jul 27 21:49:16.747021 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.747000    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.752266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.752244    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 27 21:49:16.752977 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.752954    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 27 21:49:16.762923 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.762890    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z755f"]
Jul 27 21:49:16.786973 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.786937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.787151 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.787056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9w5\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-kube-api-access-sn9w5\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.888340 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.888299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9w5\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-kube-api-access-sn9w5\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.888340 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.888343    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.902620 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.902585    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:16.907907 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:16.907887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9w5\" (UniqueName: \"kubernetes.io/projected/78a9e46a-ce5c-43c8-a1ba-9375e95752b5-kube-api-access-sn9w5\") pod \"cert-manager-cainjector-559cdd8c87-z755f\" (UID: \"78a9e46a-ce5c-43c8-a1ba-9375e95752b5\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:17.062696 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:17.062656    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f"
Jul 27 21:49:17.246624 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:17.246561    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z755f"]
Jul 27 21:49:17.249663 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:49:17.249633    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78a9e46a_ce5c_43c8_a1ba_9375e95752b5.slice/crio-47d12908fa8e21d8c7f78516de1ffd002ecf7f696772042c8abc423792ed4416 WatchSource:0}: Error finding container 47d12908fa8e21d8c7f78516de1ffd002ecf7f696772042c8abc423792ed4416: Status 404 returned error can't find the container with id 47d12908fa8e21d8c7f78516de1ffd002ecf7f696772042c8abc423792ed4416
Jul 27 21:49:18.126975 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:18.126932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f" event={"ID":"78a9e46a-ce5c-43c8-a1ba-9375e95752b5","Type":"ContainerStarted","Data":"47d12908fa8e21d8c7f78516de1ffd002ecf7f696772042c8abc423792ed4416"}
Jul 27 21:49:21.142533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:21.142499    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f" event={"ID":"78a9e46a-ce5c-43c8-a1ba-9375e95752b5","Type":"ContainerStarted","Data":"012d4c4c659d15cebfb985eab10bfe466723d45fdb06e12cb68293e5e9cf1f3c"}
Jul 27 21:49:21.204047 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:21.203991    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z755f" podStartSLOduration=2.030580692 podStartE2EDuration="5.203975739s" totalImagesPullingTime="3.173395047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:21.203196969 +0000 UTC m=+226.856625611" watchObservedRunningTime="2026-07-27 21:49:21.203975739 +0000 UTC m=+226.857404380"
Jul 27 21:49:23.769443 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.769407    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhbxt"]
Jul 27 21:49:23.772617 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.772600    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.776164 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.776149    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-h7zq7\""
Jul 27 21:49:23.791086 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.791065    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhbxt"]
Jul 27 21:49:23.850912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.850874    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb2kk\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-kube-api-access-vb2kk\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.851084 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.850924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.951557 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.951499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb2kk\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-kube-api-access-vb2kk\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.951557 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.951569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.962818 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.962793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:23.963902 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:23.963884    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb2kk\" (UniqueName: \"kubernetes.io/projected/8dfaac65-fb01-499a-a78c-df30575cdf6f-kube-api-access-vb2kk\") pod \"cert-manager-55b4d954d5-fhbxt\" (UID: \"8dfaac65-fb01-499a-a78c-df30575cdf6f\") " pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:24.081250 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:24.081223    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fhbxt"
Jul 27 21:49:24.241874 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:24.241835    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fhbxt"]
Jul 27 21:49:24.246253 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:49:24.246227    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dfaac65_fb01_499a_a78c_df30575cdf6f.slice/crio-ff657730f4e2d08208bbd5a58655ece7efa790dec910f9284478cc67da6a794f WatchSource:0}: Error finding container ff657730f4e2d08208bbd5a58655ece7efa790dec910f9284478cc67da6a794f: Status 404 returned error can't find the container with id ff657730f4e2d08208bbd5a58655ece7efa790dec910f9284478cc67da6a794f
Jul 27 21:49:25.162015 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:25.161977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fhbxt" event={"ID":"8dfaac65-fb01-499a-a78c-df30575cdf6f","Type":"ContainerStarted","Data":"2dd5fa34becf9dd32610713fa863eae446b9feb9ddba509b5cc9452ed6ee9c2c"}
Jul 27 21:49:25.162391 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:25.162026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fhbxt" event={"ID":"8dfaac65-fb01-499a-a78c-df30575cdf6f","Type":"ContainerStarted","Data":"ff657730f4e2d08208bbd5a58655ece7efa790dec910f9284478cc67da6a794f"}
Jul 27 21:49:25.189132 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:25.189076    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fhbxt" podStartSLOduration=2.189057758 podStartE2EDuration="2.189057758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:25.187893366 +0000 UTC m=+230.841322009" watchObservedRunningTime="2026-07-27 21:49:25.189057758 +0000 UTC m=+230.842486401"
Jul 27 21:49:45.883672 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.883637    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"]
Jul 27 21:49:45.888351 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.888334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:45.892054 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.892012    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 27 21:49:45.893042 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.893013    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 27 21:49:45.893042 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.893037    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 27 21:49:45.893207 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.893143    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 27 21:49:45.893409 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.893395    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jbjdz\""
Jul 27 21:49:45.920429 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.920399    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"]
Jul 27 21:49:45.946187 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.946157    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:45.946187 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.946188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:45.946360 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.946211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk859\" (UniqueName: \"kubernetes.io/projected/43afca92-4a6e-4d8b-8185-1042836ea25e-kube-api-access-jk859\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:45.946360 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.946326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:45.946430 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:45.946381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.047755 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.047714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.047912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.047773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.047912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.047792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.047912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.047821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk859\" (UniqueName: \"kubernetes.io/projected/43afca92-4a6e-4d8b-8185-1042836ea25e-kube-api-access-jk859\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.047912 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.047886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.048166 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.048136    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-manifests\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.050368 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.050337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43afca92-4a6e-4d8b-8185-1042836ea25e-tmp\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.050539 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.050517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.050638 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.050623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/43afca92-4a6e-4d8b-8185-1042836ea25e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.057640 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.057612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk859\" (UniqueName: \"kubernetes.io/projected/43afca92-4a6e-4d8b-8185-1042836ea25e-kube-api-access-jk859\") pod \"opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2\" (UID: \"43afca92-4a6e-4d8b-8185-1042836ea25e\") " pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.199693 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.199597    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:46.385477 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:46.385429    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"]
Jul 27 21:49:46.389233 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:49:46.389206    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43afca92_4a6e_4d8b_8185_1042836ea25e.slice/crio-cd858df73d563d28c9148b356099543a15ed828f261b8889d438f1b1455c13ab WatchSource:0}: Error finding container cd858df73d563d28c9148b356099543a15ed828f261b8889d438f1b1455c13ab: Status 404 returned error can't find the container with id cd858df73d563d28c9148b356099543a15ed828f261b8889d438f1b1455c13ab
Jul 27 21:49:47.260806 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:47.260763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2" event={"ID":"43afca92-4a6e-4d8b-8185-1042836ea25e","Type":"ContainerStarted","Data":"cd858df73d563d28c9148b356099543a15ed828f261b8889d438f1b1455c13ab"}
Jul 27 21:49:50.276424 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:50.276384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2" event={"ID":"43afca92-4a6e-4d8b-8185-1042836ea25e","Type":"ContainerStarted","Data":"c724a25ea6458b06d1591fea479d226720741050d51648180584d7e1a277b630"}
Jul 27 21:49:51.282812 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:51.282771    2560 generic.go:356] "Generic (PLEG): container finished" podID="43afca92-4a6e-4d8b-8185-1042836ea25e" containerID="c724a25ea6458b06d1591fea479d226720741050d51648180584d7e1a277b630" exitCode=0
Jul 27 21:49:51.283247 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:51.282854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2" event={"ID":"43afca92-4a6e-4d8b-8185-1042836ea25e","Type":"ContainerDied","Data":"c724a25ea6458b06d1591fea479d226720741050d51648180584d7e1a277b630"}
Jul 27 21:49:52.289981 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:52.289942    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2" event={"ID":"43afca92-4a6e-4d8b-8185-1042836ea25e","Type":"ContainerStarted","Data":"2ee940d9d03d8a2c79767eb4c7eef81864ca0238dc4e3d561ac3ffab2cde8e86"}
Jul 27 21:49:52.290394 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:52.290066    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:49:52.334438 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:49:52.334386    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2" podStartSLOduration=2.931935569 podStartE2EDuration="7.334364328s" totalImagesPullingTime="3.360998649s" totalInitContainerRuntime="1.04143011s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:49:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:49:52.332250347 +0000 UTC m=+257.985679008" watchObservedRunningTime="2026-07-27 21:49:52.334364328 +0000 UTC m=+257.987792971"
Jul 27 21:50:03.296193 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:03.296117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2"
Jul 27 21:50:05.472568 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.472526    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"]
Jul 27 21:50:05.479578 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.479547    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.483343 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.483318    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 27 21:50:05.483757 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.483733    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 27 21:50:05.484686 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.484666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 27 21:50:05.484779 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.484670    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 27 21:50:05.484779 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.484693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 27 21:50:05.485081 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.485060    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rpm89\""
Jul 27 21:50:05.501149 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.501116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"]
Jul 27 21:50:05.530589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.530556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e21ffd1a-601c-4740-b5df-5858b00e2df4-manager-config\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.530779 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.530596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4gm\" (UniqueName: \"kubernetes.io/projected/e21ffd1a-601c-4740-b5df-5858b00e2df4-kube-api-access-mv4gm\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.530779 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.530628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.530779 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.530716    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.631280 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.631240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.631530 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.631311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.631530 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.631407    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e21ffd1a-601c-4740-b5df-5858b00e2df4-manager-config\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.631530 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.631444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4gm\" (UniqueName: \"kubernetes.io/projected/e21ffd1a-601c-4740-b5df-5858b00e2df4-kube-api-access-mv4gm\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.632075 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.632053    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e21ffd1a-601c-4740-b5df-5858b00e2df4-manager-config\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.633849 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.633827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-metrics-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.633943 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.633881    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e21ffd1a-601c-4740-b5df-5858b00e2df4-cert\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.648564 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.648538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4gm\" (UniqueName: \"kubernetes.io/projected/e21ffd1a-601c-4740-b5df-5858b00e2df4-kube-api-access-mv4gm\") pod \"lws-controller-manager-647fb87ffb-4hjcg\" (UID: \"e21ffd1a-601c-4740-b5df-5858b00e2df4\") " pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.790058 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.789969    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:05.931595 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:05.931545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"]
Jul 27 21:50:05.933969 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:50:05.933943    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21ffd1a_601c_4740_b5df_5858b00e2df4.slice/crio-c2e5ba6ca92d22249d9212585752aa3f55ab73d2d3e2c3a9601ebd37df4569be WatchSource:0}: Error finding container c2e5ba6ca92d22249d9212585752aa3f55ab73d2d3e2c3a9601ebd37df4569be: Status 404 returned error can't find the container with id c2e5ba6ca92d22249d9212585752aa3f55ab73d2d3e2c3a9601ebd37df4569be
Jul 27 21:50:06.356088 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:06.356048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg" event={"ID":"e21ffd1a-601c-4740-b5df-5858b00e2df4","Type":"ContainerStarted","Data":"c2e5ba6ca92d22249d9212585752aa3f55ab73d2d3e2c3a9601ebd37df4569be"}
Jul 27 21:50:09.372627 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:09.372587    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg" event={"ID":"e21ffd1a-601c-4740-b5df-5858b00e2df4","Type":"ContainerStarted","Data":"1a5c5666b0285140e897d616623f40aef8056d982da6a6ced0e021e872a782bb"}
Jul 27 21:50:09.373099 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:09.372721    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:09.392814 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:09.392749    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg" podStartSLOduration=1.5713193460000001 podStartE2EDuration="4.39273435s" totalImagesPullingTime="2.821415004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:09.39051926 +0000 UTC m=+275.043947902" watchObservedRunningTime="2026-07-27 21:50:09.39273435 +0000 UTC m=+275.046162992"
Jul 27 21:50:16.060433 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.060395    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"]
Jul 27 21:50:16.064397 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.064369    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.067220 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.067189    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 27 21:50:16.067402 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.067244    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 27 21:50:16.067533 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.067515    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 21:50:16.067682 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.067662    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 21:50:16.068229 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.068215    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-llrkm\""
Jul 27 21:50:16.078425 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.078394    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"]
Jul 27 21:50:16.120641 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.120601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa1ed78-4588-47b9-9f87-56f833de56bd-tmp\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.120839 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.120703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tb9r\" (UniqueName: \"kubernetes.io/projected/6aa1ed78-4588-47b9-9f87-56f833de56bd-kube-api-access-8tb9r\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.120839 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.120754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.221932 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.221893    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa1ed78-4588-47b9-9f87-56f833de56bd-tmp\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.222145 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.221954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tb9r\" (UniqueName: \"kubernetes.io/projected/6aa1ed78-4588-47b9-9f87-56f833de56bd-kube-api-access-8tb9r\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.222145 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.222009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.222266 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:50:16.222174    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.222266 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:50:16.222253    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs podName:6aa1ed78-4588-47b9-9f87-56f833de56bd nodeName:}" failed. No retries permitted until 2026-07-27 21:50:16.722234871 +0000 UTC m=+282.375663491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs") pod "kube-auth-proxy-5794464b4b-xt64t" (UID: "6aa1ed78-4588-47b9-9f87-56f833de56bd") : secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.224474 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.224417    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa1ed78-4588-47b9-9f87-56f833de56bd-tmp\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.243649 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.243608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tb9r\" (UniqueName: \"kubernetes.io/projected/6aa1ed78-4588-47b9-9f87-56f833de56bd-kube-api-access-8tb9r\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.727383 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:16.727330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:16.727618 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:50:16.727493    2560 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 27 21:50:16.727618 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:50:16.727559    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs podName:6aa1ed78-4588-47b9-9f87-56f833de56bd nodeName:}" failed. No retries permitted until 2026-07-27 21:50:17.727544071 +0000 UTC m=+283.380972691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs") pod "kube-auth-proxy-5794464b4b-xt64t" (UID: "6aa1ed78-4588-47b9-9f87-56f833de56bd") : secret "kube-auth-proxy-tls" not found
Jul 27 21:50:17.737587 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:17.737549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:17.740157 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:17.740137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aa1ed78-4588-47b9-9f87-56f833de56bd-tls-certs\") pod \"kube-auth-proxy-5794464b4b-xt64t\" (UID: \"6aa1ed78-4588-47b9-9f87-56f833de56bd\") " pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:17.877630 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:17.877595    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"
Jul 27 21:50:18.025075 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:18.025049    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5794464b4b-xt64t"]
Jul 27 21:50:18.027000 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:50:18.026974    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa1ed78_4588_47b9_9f87_56f833de56bd.slice/crio-533ffa3607db8db05f3d3ad20676064f989ac874eba899efc2dd415152aac177 WatchSource:0}: Error finding container 533ffa3607db8db05f3d3ad20676064f989ac874eba899efc2dd415152aac177: Status 404 returned error can't find the container with id 533ffa3607db8db05f3d3ad20676064f989ac874eba899efc2dd415152aac177
Jul 27 21:50:18.428023 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:18.427984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t" event={"ID":"6aa1ed78-4588-47b9-9f87-56f833de56bd","Type":"ContainerStarted","Data":"533ffa3607db8db05f3d3ad20676064f989ac874eba899efc2dd415152aac177"}
Jul 27 21:50:20.381477 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:20.380366    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-647fb87ffb-4hjcg"
Jul 27 21:50:21.447434 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:21.447327    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t" event={"ID":"6aa1ed78-4588-47b9-9f87-56f833de56bd","Type":"ContainerStarted","Data":"9ee6a49ba1785c3a9398fd31c3e65aa959589a5a1b3743ffd32b35ac898ac31f"}
Jul 27 21:50:21.476771 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:21.476713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5794464b4b-xt64t" podStartSLOduration=2.433801463 podStartE2EDuration="5.476695461s" totalImagesPullingTime="3.042893998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:50:21.47576587 +0000 UTC m=+287.129194505" watchObservedRunningTime="2026-07-27 21:50:21.476695461 +0000 UTC m=+287.130124103"
Jul 27 21:50:34.728346 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:50:34.728320    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 21:51:50.027322 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.027235    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"]
Jul 27 21:51:50.031146 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.031124    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.033979 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.033958    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 21:51:50.034091 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.034073    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 21:51:50.034160 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.034078    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lv7rp\""
Jul 27 21:51:50.045860 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.045834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"]
Jul 27 21:51:50.106413 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.106370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vx7r\" (UniqueName: \"kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.106413 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.106414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.208086 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.208047    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vx7r\" (UniqueName: \"kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.208278 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.208092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.208555 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.208531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.216960 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.216939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vx7r\" (UniqueName: \"kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.341757 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.341716    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:50.477971 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.477855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"]
Jul 27 21:51:50.483493 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:51:50.483434    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c3410c2_9135_497f_bace_1c64dabbd4f0.slice/crio-c33f8f7f6e28fd16bfda680c10a1bcf184eba1e75d8528c04f8c2a3f2e10c36c WatchSource:0}: Error finding container c33f8f7f6e28fd16bfda680c10a1bcf184eba1e75d8528c04f8c2a3f2e10c36c: Status 404 returned error can't find the container with id c33f8f7f6e28fd16bfda680c10a1bcf184eba1e75d8528c04f8c2a3f2e10c36c
Jul 27 21:51:50.486618 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.486601    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 21:51:50.866666 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:50.866634    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp" event={"ID":"5c3410c2-9135-497f-bace-1c64dabbd4f0","Type":"ContainerStarted","Data":"c33f8f7f6e28fd16bfda680c10a1bcf184eba1e75d8528c04f8c2a3f2e10c36c"}
Jul 27 21:51:55.852589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.852545    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b8dff4fb6-hj7jd"]
Jul 27 21:51:55.856660 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.856633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.870506 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.870475    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8dff4fb6-hj7jd"]
Jul 27 21:51:55.965191 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965380 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-oauth-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965380 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-oauth-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965513 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-console-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965513 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-service-ca\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965513 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965493    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-trusted-ca-bundle\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:55.965513 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:55.965513    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4szq\" (UniqueName: \"kubernetes.io/projected/b73db49d-c709-4fa2-821d-6ac1196923cf-kube-api-access-f4szq\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066416 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-oauth-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-oauth-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-console-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066639 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066633    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-service-ca\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066867 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066659    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-trusted-ca-bundle\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.066867 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.066684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4szq\" (UniqueName: \"kubernetes.io/projected/b73db49d-c709-4fa2-821d-6ac1196923cf-kube-api-access-f4szq\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.067881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.067249    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-oauth-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.067881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.067258    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-console-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.067881 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.067834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-service-ca\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.068121 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.067953    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b73db49d-c709-4fa2-821d-6ac1196923cf-trusted-ca-bundle\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.069587 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.069544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-oauth-config\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.069587 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.069555    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b73db49d-c709-4fa2-821d-6ac1196923cf-console-serving-cert\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.081811 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.081784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4szq\" (UniqueName: \"kubernetes.io/projected/b73db49d-c709-4fa2-821d-6ac1196923cf-kube-api-access-f4szq\") pod \"console-5b8dff4fb6-hj7jd\" (UID: \"b73db49d-c709-4fa2-821d-6ac1196923cf\") " pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.169054 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.168967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:51:56.668337 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:51:56.668310    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73db49d_c709_4fa2_821d_6ac1196923cf.slice/crio-b5ed94be53c39b5e685702772d1a8ecd169fd7270ccf03471a578d8c42a1e6a1 WatchSource:0}: Error finding container b5ed94be53c39b5e685702772d1a8ecd169fd7270ccf03471a578d8c42a1e6a1: Status 404 returned error can't find the container with id b5ed94be53c39b5e685702772d1a8ecd169fd7270ccf03471a578d8c42a1e6a1
Jul 27 21:51:56.668899 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.668812    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b8dff4fb6-hj7jd"]
Jul 27 21:51:56.905587 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.905546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp" event={"ID":"5c3410c2-9135-497f-bace-1c64dabbd4f0","Type":"ContainerStarted","Data":"4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014"}
Jul 27 21:51:56.906245 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.905647    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:51:56.907520 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.907492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8dff4fb6-hj7jd" event={"ID":"b73db49d-c709-4fa2-821d-6ac1196923cf","Type":"ContainerStarted","Data":"ab10dfd7e2fbc172993af19d01af96bd4bf38ce5bd821e0f88d93423f3fd5193"}
Jul 27 21:51:56.907643 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.907534    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b8dff4fb6-hj7jd" event={"ID":"b73db49d-c709-4fa2-821d-6ac1196923cf","Type":"ContainerStarted","Data":"b5ed94be53c39b5e685702772d1a8ecd169fd7270ccf03471a578d8c42a1e6a1"}
Jul 27 21:51:56.938822 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.938763    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp" podStartSLOduration=0.843267084 podStartE2EDuration="6.938747022s" totalImagesPullingTime="6.095479938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:56.935530295 +0000 UTC m=+382.588958955" watchObservedRunningTime="2026-07-27 21:51:56.938747022 +0000 UTC m=+382.592175664"
Jul 27 21:51:56.973614 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:51:56.973553    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b8dff4fb6-hj7jd" podStartSLOduration=1.973534393 podStartE2EDuration="1.973534393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:51:56.971169736 +0000 UTC m=+382.624598377" watchObservedRunningTime="2026-07-27 21:51:56.973534393 +0000 UTC m=+382.626963035"
Jul 27 21:52:06.169683 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:06.169645    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:52:06.170124 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:06.169719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:52:06.174307 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:06.174284    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:52:06.962625 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:06.962596    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b8dff4fb6-hj7jd"
Jul 27 21:52:07.017822 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:07.017783    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:52:07.914681 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:07.914652    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.675323 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.675286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"]
Jul 27 21:52:09.675751 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.675497    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" containerName="manager" containerID="cri-o://4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014" gracePeriod=2
Jul 27 21:52:09.692283 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.692251    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"]
Jul 27 21:52:09.703678 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.703646    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:09.704019 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.704005    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" containerName="manager"
Jul 27 21:52:09.704019 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.704019    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" containerName="manager"
Jul 27 21:52:09.704138 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.704080    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" containerName="manager"
Jul 27 21:52:09.707074 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.707056    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.710507 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:09.710478    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" is forbidden: User \"system:node:ip-10-0-143-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-110.ec2.internal' and this object" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.728966 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.728933    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:09.796716 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.796687    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wwm\" (UniqueName: \"kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.796855 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.796758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.897564 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.897530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wwm\" (UniqueName: \"kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.897738 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.897629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.898003 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.897984    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.909489 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.909465    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wwm\" (UniqueName: \"kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hftt9\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:09.928834 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.928779    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.931065 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:09.931036    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" is forbidden: User \"system:node:ip-10-0-143-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-110.ec2.internal' and this object" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.983220 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.983165    2560 generic.go:356] "Generic (PLEG): container finished" podID="5c3410c2-9135-497f-bace-1c64dabbd4f0" containerID="4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014" exitCode=0
Jul 27 21:52:09.983220 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.983228    2560 scope.go:117] "RemoveContainer" containerID="4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014"
Jul 27 21:52:09.983523 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.983238    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.985939 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:09.985907    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" is forbidden: User \"system:node:ip-10-0-143-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-110.ec2.internal' and this object" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:09.992095 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.992076    2560 scope.go:117] "RemoveContainer" containerID="4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014"
Jul 27 21:52:09.992349 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:09.992330    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014\": container with ID starting with 4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014 not found: ID does not exist" containerID="4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014"
Jul 27 21:52:09.992417 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.992362    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014"} err="failed to get container status \"4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014\": rpc error: code = NotFound desc = could not find container \"4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014\": container with ID starting with 4ffe9c7302f75bb8ffbbfcc36bda034b72caeda0d2e5c3dbe9ca1dd726123014 not found: ID does not exist"
Jul 27 21:52:09.998674 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.998655    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vx7r\" (UniqueName: \"kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r\") pod \"5c3410c2-9135-497f-bace-1c64dabbd4f0\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") "
Jul 27 21:52:09.998754 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.998687    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume\") pod \"5c3410c2-9135-497f-bace-1c64dabbd4f0\" (UID: \"5c3410c2-9135-497f-bace-1c64dabbd4f0\") "
Jul 27 21:52:09.999212 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:09.999189    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5c3410c2-9135-497f-bace-1c64dabbd4f0" (UID: "5c3410c2-9135-497f-bace-1c64dabbd4f0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:52:10.000844 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.000826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r" (OuterVolumeSpecName: "kube-api-access-7vx7r") pod "5c3410c2-9135-497f-bace-1c64dabbd4f0" (UID: "5c3410c2-9135-497f-bace-1c64dabbd4f0"). InnerVolumeSpecName "kube-api-access-7vx7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:10.054072 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.054039    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:10.099555 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.099520    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vx7r\" (UniqueName: \"kubernetes.io/projected/5c3410c2-9135-497f-bace-1c64dabbd4f0-kube-api-access-7vx7r\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:10.099555 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.099552    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c3410c2-9135-497f-bace-1c64dabbd4f0-extensions-socket-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:10.200339 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.200310    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:10.202692 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:52:10.202670    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b458e0_a6ef_4d13_a41c_6d0cd8bf31e4.slice/crio-1587cb3bb0786591545e98952ecbb385a123ccaf236e22abf791e75d24fcb425 WatchSource:0}: Error finding container 1587cb3bb0786591545e98952ecbb385a123ccaf236e22abf791e75d24fcb425: Status 404 returned error can't find the container with id 1587cb3bb0786591545e98952ecbb385a123ccaf236e22abf791e75d24fcb425
Jul 27 21:52:10.293968 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:10.293936    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-wcbtp\" is forbidden: User \"system:node:ip-10-0-143-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-110.ec2.internal' and this object" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcbtp"
Jul 27 21:52:10.915673 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.915633    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c3410c2-9135-497f-bace-1c64dabbd4f0" path="/var/lib/kubelet/pods/5c3410c2-9135-497f-bace-1c64dabbd4f0/volumes"
Jul 27 21:52:10.989971 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.989937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" event={"ID":"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4","Type":"ContainerStarted","Data":"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"}
Jul 27 21:52:10.990159 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.989988    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" event={"ID":"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4","Type":"ContainerStarted","Data":"1587cb3bb0786591545e98952ecbb385a123ccaf236e22abf791e75d24fcb425"}
Jul 27 21:52:10.990159 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:10.990035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:11.013137 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:11.013084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" podStartSLOduration=2.013067482 podStartE2EDuration="2.013067482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:11.011911081 +0000 UTC m=+396.665339723" watchObservedRunningTime="2026-07-27 21:52:11.013067482 +0000 UTC m=+396.666496124"
Jul 27 21:52:21.998956 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:21.998924    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:25.735484 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:25.735436    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:25.735910 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:25.735631    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" podUID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" containerName="manager" containerID="cri-o://2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393" gracePeriod=10
Jul 27 21:52:25.985987 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:25.985920    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:26.048436 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.048401    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5wwm\" (UniqueName: \"kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm\") pod \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") "
Jul 27 21:52:26.048436 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.048443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume\") pod \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\" (UID: \"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4\") "
Jul 27 21:52:26.048930 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.048901    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" (UID: "72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 21:52:26.050879 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.050856    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm" (OuterVolumeSpecName: "kube-api-access-z5wwm") pod "72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" (UID: "72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4"). InnerVolumeSpecName "kube-api-access-z5wwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:26.065168 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.065143    2560 generic.go:356] "Generic (PLEG): container finished" podID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" containerID="2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393" exitCode=0
Jul 27 21:52:26.065266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.065190    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"
Jul 27 21:52:26.065266 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.065236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" event={"ID":"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4","Type":"ContainerDied","Data":"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"}
Jul 27 21:52:26.065340 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.065287    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9" event={"ID":"72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4","Type":"ContainerDied","Data":"1587cb3bb0786591545e98952ecbb385a123ccaf236e22abf791e75d24fcb425"}
Jul 27 21:52:26.065340 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.065310    2560 scope.go:117] "RemoveContainer" containerID="2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"
Jul 27 21:52:26.074647 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.074623    2560 scope.go:117] "RemoveContainer" containerID="2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"
Jul 27 21:52:26.074919 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:26.074900    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393\": container with ID starting with 2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393 not found: ID does not exist" containerID="2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"
Jul 27 21:52:26.074963 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.074928    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393"} err="failed to get container status \"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393\": rpc error: code = NotFound desc = could not find container \"2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393\": container with ID starting with 2d751876554674588d749cf8601561c1f640ab9fbb814e56dfe966a1de3be393 not found: ID does not exist"
Jul 27 21:52:26.095182 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.095147    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:26.106887 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.106858    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 21:52:26.107283 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.107266    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" containerName="manager"
Jul 27 21:52:26.107283 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.107283    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" containerName="manager"
Jul 27 21:52:26.107462 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.107348    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" containerName="manager"
Jul 27 21:52:26.111598 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.111579    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hftt9"]
Jul 27 21:52:26.111765 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.111665    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.114289 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.114269    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 21:52:26.114386 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.114271    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 21:52:26.114386 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.114304    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lv7rp\""
Jul 27 21:52:26.123878 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.123857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 21:52:26.150361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.150325    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wwm\" (UniqueName: \"kubernetes.io/projected/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-kube-api-access-z5wwm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:26.150361 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.150356    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4-extensions-socket-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:26.251344 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.251254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.251344 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.251313    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86p87\" (UniqueName: \"kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.352332 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.352293    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86p87\" (UniqueName: \"kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.352538 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.352389    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.352750 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.352732    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.363706 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.363678    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86p87\" (UniqueName: \"kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87\") pod \"kuadrant-operator-controller-manager-86878c769f-6sb4n\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.421835 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.421796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:26.562232 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.562187    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 21:52:26.564835 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:52:26.564807    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7bf442_de23_4330_b018_4d90643cf611.slice/crio-1c55ccde6091f2872b7529080f83a99edec4ffc25360eaeec6e4e42047789b2d WatchSource:0}: Error finding container 1c55ccde6091f2872b7529080f83a99edec4ffc25360eaeec6e4e42047789b2d: Status 404 returned error can't find the container with id 1c55ccde6091f2872b7529080f83a99edec4ffc25360eaeec6e4e42047789b2d
Jul 27 21:52:26.915264 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:26.915226    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4" path="/var/lib/kubelet/pods/72b458e0-a6ef-4d13-a41c-6d0cd8bf31e4/volumes"
Jul 27 21:52:27.073292 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:27.073256    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" event={"ID":"ef7bf442-de23-4330-b018-4d90643cf611","Type":"ContainerStarted","Data":"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"}
Jul 27 21:52:27.073532 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:27.073306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" event={"ID":"ef7bf442-de23-4330-b018-4d90643cf611","Type":"ContainerStarted","Data":"1c55ccde6091f2872b7529080f83a99edec4ffc25360eaeec6e4e42047789b2d"}
Jul 27 21:52:27.073532 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:27.073401    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:27.102025 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:27.101961    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" podStartSLOduration=1.101941546 podStartE2EDuration="1.101941546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:52:27.10169213 +0000 UTC m=+412.755120772" watchObservedRunningTime="2026-07-27 21:52:27.101941546 +0000 UTC m=+412.755370199"
Jul 27 21:52:32.039977 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.039936    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55b55cf6b6-sgxrj" podUID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" containerName="console" containerID="cri-o://133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad" gracePeriod=15
Jul 27 21:52:32.290767 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.290707    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:52:32.411818 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4hsg\" (UniqueName: \"kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.411997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411839    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.411997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411867    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.411997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411896    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.411997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411935    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.411997 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.411990    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.412229 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.412039    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config\") pod \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\" (UID: \"1da729cd-43d0-4df3-bf1a-d8b7a7355742\") "
Jul 27 21:52:32.412445 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.412408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config" (OuterVolumeSpecName: "console-config") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:32.412445 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.412420    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:32.412812 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.412431    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:32.412812 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.412564    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca" (OuterVolumeSpecName: "service-ca") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 21:52:32.414224 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.414198    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:52:32.414368 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.414266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg" (OuterVolumeSpecName: "kube-api-access-d4hsg") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "kube-api-access-d4hsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 21:52:32.414426 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.414411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1da729cd-43d0-4df3-bf1a-d8b7a7355742" (UID: "1da729cd-43d0-4df3-bf1a-d8b7a7355742"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 21:52:32.513006 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.512969    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hsg\" (UniqueName: \"kubernetes.io/projected/1da729cd-43d0-4df3-bf1a-d8b7a7355742-kube-api-access-d4hsg\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513006 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513001    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-oauth-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513006 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513012    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513249 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513021    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-serving-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513249 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513040    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-trusted-ca-bundle\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513249 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513049    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1da729cd-43d0-4df3-bf1a-d8b7a7355742-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:32.513249 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:32.513057    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1da729cd-43d0-4df3-bf1a-d8b7a7355742-console-oauth-config\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 21:52:33.102954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.102915    2560 generic.go:356] "Generic (PLEG): container finished" podID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" containerID="133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad" exitCode=0
Jul 27 21:52:33.103439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.102960    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55b55cf6b6-sgxrj"
Jul 27 21:52:33.103439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.102979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b55cf6b6-sgxrj" event={"ID":"1da729cd-43d0-4df3-bf1a-d8b7a7355742","Type":"ContainerDied","Data":"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"}
Jul 27 21:52:33.103439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.103032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55b55cf6b6-sgxrj" event={"ID":"1da729cd-43d0-4df3-bf1a-d8b7a7355742","Type":"ContainerDied","Data":"60726d6f8c4c4ef99921b5deed72d232e2826e14c8bc3337b8ffe660dae39701"}
Jul 27 21:52:33.103439 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.103061    2560 scope.go:117] "RemoveContainer" containerID="133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"
Jul 27 21:52:33.115073 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.115055    2560 scope.go:117] "RemoveContainer" containerID="133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"
Jul 27 21:52:33.115352 ip-10-0-143-110 kubenswrapper[2560]: E0727 21:52:33.115331    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad\": container with ID starting with 133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad not found: ID does not exist" containerID="133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"
Jul 27 21:52:33.115408 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.115361    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad"} err="failed to get container status \"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad\": rpc error: code = NotFound desc = could not find container \"133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad\": container with ID starting with 133c9e39d7e6f061d85736c3d374a0bb074fcd45809e9485bdd86727467266ad not found: ID does not exist"
Jul 27 21:52:33.135753 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.135722    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:52:33.146902 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:33.146876    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55b55cf6b6-sgxrj"]
Jul 27 21:52:34.915669 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:34.915637    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" path="/var/lib/kubelet/pods/1da729cd-43d0-4df3-bf1a-d8b7a7355742/volumes"
Jul 27 21:52:38.080192 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:38.080163    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 21:52:56.451436 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.451349    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:52:56.451861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.451776    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" containerName="console"
Jul 27 21:52:56.451861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.451788    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" containerName="console"
Jul 27 21:52:56.451861 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.451843    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1da729cd-43d0-4df3-bf1a-d8b7a7355742" containerName="console"
Jul 27 21:52:56.454884 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.454864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.457444 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.457420    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-r79nw\""
Jul 27 21:52:56.457568 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.457445    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 21:52:56.464972 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.464944    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:52:56.510567 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.510535    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:52:56.537029 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.536991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86bdeaa-35c4-4785-99dd-da371ff88db6-config-file\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.537215 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.537151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq59r\" (UniqueName: \"kubernetes.io/projected/d86bdeaa-35c4-4785-99dd-da371ff88db6-kube-api-access-vq59r\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.638602 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.638564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq59r\" (UniqueName: \"kubernetes.io/projected/d86bdeaa-35c4-4785-99dd-da371ff88db6-kube-api-access-vq59r\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.638790 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.638616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86bdeaa-35c4-4785-99dd-da371ff88db6-config-file\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.639263 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.639244    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d86bdeaa-35c4-4785-99dd-da371ff88db6-config-file\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.647723 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.647691    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq59r\" (UniqueName: \"kubernetes.io/projected/d86bdeaa-35c4-4785-99dd-da371ff88db6-kube-api-access-vq59r\") pod \"limitador-limitador-54d6d988d5-zqts9\" (UID: \"d86bdeaa-35c4-4785-99dd-da371ff88db6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.765545 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.765434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:52:56.901344 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:56.901270    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:52:56.903678 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:52:56.903648    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86bdeaa_35c4_4785_99dd_da371ff88db6.slice/crio-4b1a0b585ff6620d4d5f145bcdfe1e5dfe4e9c2c63c508eb3481f3a24a7473e0 WatchSource:0}: Error finding container 4b1a0b585ff6620d4d5f145bcdfe1e5dfe4e9c2c63c508eb3481f3a24a7473e0: Status 404 returned error can't find the container with id 4b1a0b585ff6620d4d5f145bcdfe1e5dfe4e9c2c63c508eb3481f3a24a7473e0
Jul 27 21:52:57.214002 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:52:57.213967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9" event={"ID":"d86bdeaa-35c4-4785-99dd-da371ff88db6","Type":"ContainerStarted","Data":"4b1a0b585ff6620d4d5f145bcdfe1e5dfe4e9c2c63c508eb3481f3a24a7473e0"}
Jul 27 21:53:00.231518 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:53:00.231438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9" event={"ID":"d86bdeaa-35c4-4785-99dd-da371ff88db6","Type":"ContainerStarted","Data":"01d5b1ecae2df7375d50d664d931ee14eb376b79443fb8a42efbd45e45a67383"}
Jul 27 21:53:00.231898 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:53:00.231601    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:53:00.250336 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:53:00.250287    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9" podStartSLOduration=1.701998112 podStartE2EDuration="4.250271135s" totalImagesPullingTime="2.548273023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:53:00.248980152 +0000 UTC m=+445.902408827" watchObservedRunningTime="2026-07-27 21:53:00.250271135 +0000 UTC m=+445.903699777"
Jul 27 21:53:11.237830 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:53:11.237794    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-zqts9"
Jul 27 21:54:12.581421 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:54:12.581387    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:54:27.956857 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:54:27.956766    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:54:32.055165 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:54:32.055121    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:54:35.454242 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:54:35.454194    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:54:40.848210 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:54:40.848176    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:55:06.748589 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:06.748553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:55:19.645252 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.645210    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"]
Jul 27 21:55:19.649001 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.648982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.651800 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.651780    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 27 21:55:19.652833 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.652812    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 27 21:55:19.652954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.652865    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 27 21:55:19.652954 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.652882    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h4jdf\""
Jul 27 21:55:19.659806 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.659779    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"]
Jul 27 21:55:19.743978 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.743929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.744168 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.744071    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.744168 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.744125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kl8d\" (UniqueName: \"kubernetes.io/projected/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-kube-api-access-5kl8d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.744168 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.744151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.744272 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.744194    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.744272 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.744224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845080 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kl8d\" (UniqueName: \"kubernetes.io/projected/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-kube-api-access-5kl8d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845080 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845080    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845315 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845315 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845315 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845315 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845565 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845483    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845630 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.845630 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.845617    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.847583 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.847564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.847689 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.847671    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.853832 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.853811    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kl8d\" (UniqueName: \"kubernetes.io/projected/2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4-kube-api-access-5kl8d\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l\" (UID: \"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:19.958923 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:19.958826    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:20.095680 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:20.095438    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"]
Jul 27 21:55:20.098219 ip-10-0-143-110 kubenswrapper[2560]: W0727 21:55:20.098187    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2058dc0a_fc13_46f6_b4bc_7ab1c53a17a4.slice/crio-3e52d025488ada68de12fb6c1614159193784fa2b2ea605c8748e39ad700951a WatchSource:0}: Error finding container 3e52d025488ada68de12fb6c1614159193784fa2b2ea605c8748e39ad700951a: Status 404 returned error can't find the container with id 3e52d025488ada68de12fb6c1614159193784fa2b2ea605c8748e39ad700951a
Jul 27 21:55:20.896688 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:20.896649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l" event={"ID":"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4","Type":"ContainerStarted","Data":"3e52d025488ada68de12fb6c1614159193784fa2b2ea605c8748e39ad700951a"}
Jul 27 21:55:21.903862 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:21.903819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l" event={"ID":"2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4","Type":"ContainerStarted","Data":"7dc3234c372bdeb57a69d6264e1d0ab7c6b7f1f88e59c21bee095682b7ad90a4"}
Jul 27 21:55:21.923046 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:21.922961    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l" podStartSLOduration=1.744106824 podStartE2EDuration="2.922941296s" totalImagesPullingTime="1.178834472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 21:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 21:55:21.921186764 +0000 UTC m=+587.574615411" watchObservedRunningTime="2026-07-27 21:55:21.922941296 +0000 UTC m=+587.576369938"
Jul 27 21:55:22.358623 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:22.358592    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:55:29.959793 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:29.959753    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:29.960291 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:29.959904    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:29.972296 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:29.972268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:55:30.962949 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:55:30.962922    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l"
Jul 27 21:56:52.464445 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:56:52.464407    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:57:02.072230 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:57:02.072191    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:57:12.753498 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:57:12.753463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:58:55.551965 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:58:55.551875    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:59:03.857232 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:59:03.857179    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:59:35.553915 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:59:35.553876    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:59:44.951208 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:59:44.951174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 21:59:55.755944 ip-10-0-143-110 kubenswrapper[2560]: I0727 21:59:55.755905    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:00:03.656804 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:00:03.656759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:00:15.359812 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:00:15.359772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:00:25.159623 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:00:25.159521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:00:35.857063 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:00:35.857018    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:00:45.554748 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:00:45.554714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:01:08.550275 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:01:08.550234    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:01:17.470291 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:01:17.470249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:01:42.078526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:01:42.078484    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:01:52.462589 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:01:52.462552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:02:07.556540 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:02:07.556445    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:02:18.569598 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:02:18.569556    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:02:27.755842 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:02:27.755802    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:02:37.968082 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:02:37.968041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:02:47.084099 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:02:47.084056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:03:12.615584 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:12.615549    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 22:03:12.616064 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:12.615792    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" podUID="ef7bf442-de23-4330-b018-4d90643cf611" containerName="manager" containerID="cri-o://21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753" gracePeriod=10
Jul 27 22:03:15.369403 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.369375    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 22:03:15.445627 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.445549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86p87\" (UniqueName: \"kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87\") pod \"ef7bf442-de23-4330-b018-4d90643cf611\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") "
Jul 27 22:03:15.445627 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.445599    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume\") pod \"ef7bf442-de23-4330-b018-4d90643cf611\" (UID: \"ef7bf442-de23-4330-b018-4d90643cf611\") "
Jul 27 22:03:15.446009 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.445983    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ef7bf442-de23-4330-b018-4d90643cf611" (UID: "ef7bf442-de23-4330-b018-4d90643cf611"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:03:15.447791 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.447770    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87" (OuterVolumeSpecName: "kube-api-access-86p87") pod "ef7bf442-de23-4330-b018-4d90643cf611" (UID: "ef7bf442-de23-4330-b018-4d90643cf611"). InnerVolumeSpecName "kube-api-access-86p87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:03:15.546371 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.546329    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86p87\" (UniqueName: \"kubernetes.io/projected/ef7bf442-de23-4330-b018-4d90643cf611-kube-api-access-86p87\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:03:15.546371 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:15.546363    2560 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef7bf442-de23-4330-b018-4d90643cf611-extensions-socket-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:03:16.199266 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.199223    2560 generic.go:356] "Generic (PLEG): container finished" podID="ef7bf442-de23-4330-b018-4d90643cf611" containerID="21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753" exitCode=0
Jul 27 22:03:16.199266 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.199269    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"
Jul 27 22:03:16.199529 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.199307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" event={"ID":"ef7bf442-de23-4330-b018-4d90643cf611","Type":"ContainerDied","Data":"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"}
Jul 27 22:03:16.199529 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.199368    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n" event={"ID":"ef7bf442-de23-4330-b018-4d90643cf611","Type":"ContainerDied","Data":"1c55ccde6091f2872b7529080f83a99edec4ffc25360eaeec6e4e42047789b2d"}
Jul 27 22:03:16.199529 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.199390    2560 scope.go:117] "RemoveContainer" containerID="21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"
Jul 27 22:03:16.209167 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.209149    2560 scope.go:117] "RemoveContainer" containerID="21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"
Jul 27 22:03:16.209406 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:03:16.209387    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753\": container with ID starting with 21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753 not found: ID does not exist" containerID="21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"
Jul 27 22:03:16.209475 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.209415    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753"} err="failed to get container status \"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753\": rpc error: code = NotFound desc = could not find container \"21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753\": container with ID starting with 21d1d0ab910dce82baf6ef9eaf7854dbb36aa55f76da3c6e2255b4f6bae3e753 not found: ID does not exist"
Jul 27 22:03:16.227406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.227366    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 22:03:16.227642 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.227620    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6sb4n"]
Jul 27 22:03:16.916025 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:03:16.915991    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7bf442-de23-4330-b018-4d90643cf611" path="/var/lib/kubelet/pods/ef7bf442-de23-4330-b018-4d90643cf611/volumes"
Jul 27 22:04:13.831050 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.831004    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"]
Jul 27 22:04:13.831875 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.831639    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7bf442-de23-4330-b018-4d90643cf611" containerName="manager"
Jul 27 22:04:13.831875 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.831659    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7bf442-de23-4330-b018-4d90643cf611" containerName="manager"
Jul 27 22:04:13.831875 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.831773    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7bf442-de23-4330-b018-4d90643cf611" containerName="manager"
Jul 27 22:04:13.834771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.834748    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.837726 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.837701    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lv7rp\""
Jul 27 22:04:13.858765 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.858733    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"]
Jul 27 22:04:13.884796 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.884753    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dcbc993-b181-42b4-a342-c2bda2bca1ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.884971 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.884808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57nc8\" (UniqueName: \"kubernetes.io/projected/9dcbc993-b181-42b4-a342-c2bda2bca1ce-kube-api-access-57nc8\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.986387 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.986351    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dcbc993-b181-42b4-a342-c2bda2bca1ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.986619 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.986403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57nc8\" (UniqueName: \"kubernetes.io/projected/9dcbc993-b181-42b4-a342-c2bda2bca1ce-kube-api-access-57nc8\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.986830 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.986807    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9dcbc993-b181-42b4-a342-c2bda2bca1ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:13.999871 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:13.999839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57nc8\" (UniqueName: \"kubernetes.io/projected/9dcbc993-b181-42b4-a342-c2bda2bca1ce-kube-api-access-57nc8\") pod \"kuadrant-operator-controller-manager-86878c769f-g9kpl\" (UID: \"9dcbc993-b181-42b4-a342-c2bda2bca1ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:14.146178 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.146078    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:14.293436 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.293411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"]
Jul 27 22:04:14.295720 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:04:14.295696    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dcbc993_b181_42b4_a342_c2bda2bca1ce.slice/crio-6fbdc7c1bcae258e9bd47312ed161856ddc00efee4d14e545c852bada5bcbfae WatchSource:0}: Error finding container 6fbdc7c1bcae258e9bd47312ed161856ddc00efee4d14e545c852bada5bcbfae: Status 404 returned error can't find the container with id 6fbdc7c1bcae258e9bd47312ed161856ddc00efee4d14e545c852bada5bcbfae
Jul 27 22:04:14.473968 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.473861    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl" event={"ID":"9dcbc993-b181-42b4-a342-c2bda2bca1ce","Type":"ContainerStarted","Data":"b0dad16374fb5f6825c05b5199bd71ea9de22388c382cb561fc0bbc827c65a55"}
Jul 27 22:04:14.473968 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.473931    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:14.473968 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.473954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl" event={"ID":"9dcbc993-b181-42b4-a342-c2bda2bca1ce","Type":"ContainerStarted","Data":"6fbdc7c1bcae258e9bd47312ed161856ddc00efee4d14e545c852bada5bcbfae"}
Jul 27 22:04:14.497183 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:14.497128    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl" podStartSLOduration=1.4971126240000001 podStartE2EDuration="1.497112624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:04:14.494893457 +0000 UTC m=+1120.148322103" watchObservedRunningTime="2026-07-27 22:04:14.497112624 +0000 UTC m=+1120.150541244"
Jul 27 22:04:25.480728 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:25.480694    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-g9kpl"
Jul 27 22:04:35.559378 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:35.559329    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:04:46.952295 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:46.952259    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:04:57.750278 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:04:57.750201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:05:08.363781 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:05:08.363744    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:05:20.159311 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:05:20.159265    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:05:29.258136 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:05:29.258092    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:05:39.965827 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:05:39.965791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:05:49.248549 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:05:49.248512    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:06:00.953893 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:06:00.953818    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:06:10.656337 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:06:10.656295    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:06:20.665442 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:06:20.665395    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:06:32.257229 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:06:32.257147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:06:54.960852 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:06:54.960815    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:07:03.749929 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.749892    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:03.753598 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.753573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.756526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.756502    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e7af719f-kserve-self-signed-certs\""
Jul 27 22:07:03.770971 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.770943    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:03.779526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxz7\" (UniqueName: \"kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.779648 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.779703 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779660    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.779703 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.779802 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.779802 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.779789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.880973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.880930    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.880973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.880981    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881245 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881007    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxz7\" (UniqueName: \"kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881245 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881245 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881245 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881489 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881489 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.881603 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.881544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.883543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.883515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.883722 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.883705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:03.890495 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:03.890473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxz7\" (UniqueName: \"kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7\") pod \"e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") " pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:04.063549 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:04.063516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:04.211004 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:04.210956    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:04.213981 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:07:04.213936    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod401f0561_e011_4097_be1a_370f199e54a4.slice/crio-bc30037d3f30726d1b1b409450835dde46f5daf12cdc59fd824f92155d362251 WatchSource:0}: Error finding container bc30037d3f30726d1b1b409450835dde46f5daf12cdc59fd824f92155d362251: Status 404 returned error can't find the container with id bc30037d3f30726d1b1b409450835dde46f5daf12cdc59fd824f92155d362251
Jul 27 22:07:04.297484 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:04.297413    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" event={"ID":"401f0561-e011-4097-be1a-370f199e54a4","Type":"ContainerStarted","Data":"baea40c404a90cd28285221e9fda0d66e1676a5241fb1041065eba05b88d2393"}
Jul 27 22:07:04.297684 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:04.297494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" event={"ID":"401f0561-e011-4097-be1a-370f199e54a4","Type":"ContainerStarted","Data":"bc30037d3f30726d1b1b409450835dde46f5daf12cdc59fd824f92155d362251"}
Jul 27 22:07:04.316014 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:04.315895    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" podStartSLOduration=1.315878195 podStartE2EDuration="1.315878195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:07:04.313888911 +0000 UTC m=+1289.967317569" watchObservedRunningTime="2026-07-27 22:07:04.315878195 +0000 UTC m=+1289.969306837"
Jul 27 22:07:05.352950 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:05.352914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:07:14.064651 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:14.064616    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:14.065126 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:14.064671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:14.077877 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:14.077843    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:14.359430 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:14.359321    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:20.253719 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:20.253672    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:07:31.260220 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:31.260189    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:33.759721 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:33.759681    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:07:46.280220 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.280145    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" podUID="401f0561-e011-4097-be1a-370f199e54a4" containerName="main" containerID="cri-o://baea40c404a90cd28285221e9fda0d66e1676a5241fb1041065eba05b88d2393" gracePeriod=45
Jul 27 22:07:46.519794 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.519763    2560 generic.go:356] "Generic (PLEG): container finished" podID="401f0561-e011-4097-be1a-370f199e54a4" containerID="baea40c404a90cd28285221e9fda0d66e1676a5241fb1041065eba05b88d2393" exitCode=0
Jul 27 22:07:46.519943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.519833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" event={"ID":"401f0561-e011-4097-be1a-370f199e54a4","Type":"ContainerDied","Data":"baea40c404a90cd28285221e9fda0d66e1676a5241fb1041065eba05b88d2393"}
Jul 27 22:07:46.537012 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.536950    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:46.608647 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608604    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxxz7\" (UniqueName: \"kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.608841 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608681    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.608841 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608740    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.608841 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.609013 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608842    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.609013 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608908    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs\") pod \"401f0561-e011-4097-be1a-370f199e54a4\" (UID: \"401f0561-e011-4097-be1a-370f199e54a4\") "
Jul 27 22:07:46.609113 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.608987    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:07:46.609113 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.609044    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:07:46.609221 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.609203    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home" (OuterVolumeSpecName: "home") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:07:46.609300 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.609281    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:46.609356 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.609305    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:46.611224 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.611194    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm" (OuterVolumeSpecName: "dshm") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:07:46.611333 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.611261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7" (OuterVolumeSpecName: "kube-api-access-fxxz7") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "kube-api-access-fxxz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:07:46.611424 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.611410    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "401f0561-e011-4097-be1a-370f199e54a4" (UID: "401f0561-e011-4097-be1a-370f199e54a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:07:46.710516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.710477    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/401f0561-e011-4097-be1a-370f199e54a4-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:46.710516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.710509    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxz7\" (UniqueName: \"kubernetes.io/projected/401f0561-e011-4097-be1a-370f199e54a4-kube-api-access-fxxz7\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:46.710516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.710519    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:46.710516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:46.710527    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/401f0561-e011-4097-be1a-370f199e54a4-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:07:47.527407 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.527371    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"
Jul 27 22:07:47.527911 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.527409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx" event={"ID":"401f0561-e011-4097-be1a-370f199e54a4","Type":"ContainerDied","Data":"bc30037d3f30726d1b1b409450835dde46f5daf12cdc59fd824f92155d362251"}
Jul 27 22:07:47.527911 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.527477    2560 scope.go:117] "RemoveContainer" containerID="baea40c404a90cd28285221e9fda0d66e1676a5241fb1041065eba05b88d2393"
Jul 27 22:07:47.545000 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.544964    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:47.550254 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.550221    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e7af719f-kserve-665686f847-xvvfx"]
Jul 27 22:07:47.956247 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:47.956201    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:07:48.915464 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:48.915417    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401f0561-e011-4097-be1a-370f199e54a4" path="/var/lib/kubelet/pods/401f0561-e011-4097-be1a-370f199e54a4/volumes"
Jul 27 22:07:54.852727 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:07:54.852691    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:08:10.954917 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:08:10.954874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:08:25.364326 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:08:25.364278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:08:34.662248 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:08:34.662209    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:08:49.862367 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:08:49.862320    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:08:59.152674 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:08:59.152627    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:09:12.291024 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:09:12.290981    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:09:21.464875 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:09:21.464830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:09:31.268876 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:09:31.268781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:09:43.852948 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:09:43.852909    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:09:56.360663 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:09:56.360604    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:04.462549 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:04.462517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:17.758048 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:17.758000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:30.560146 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:30.560107    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:39.968232 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:39.968185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:47.460916 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:47.460869    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:10:56.359205 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:10:56.359106    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:11:18.751746 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:18.751711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:11:33.866550 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:33.866507    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:11:52.785947 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.785912    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"]
Jul 27 22:11:52.786406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.786298    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401f0561-e011-4097-be1a-370f199e54a4" containerName="main"
Jul 27 22:11:52.786406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.786310    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="401f0561-e011-4097-be1a-370f199e54a4" containerName="main"
Jul 27 22:11:52.786406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.786394    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="401f0561-e011-4097-be1a-370f199e54a4" containerName="main"
Jul 27 22:11:52.789351 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.789334    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:52.791931 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.791910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:11:52.792008 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.791936    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:11:52.796744 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.796499    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"]
Jul 27 22:11:52.967023 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:52.966989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66e79c4-r6597\" (UID: \"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:53.068567 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.068537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66e79c4-r6597\" (UID: \"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:53.069165 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.069137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66e79c4-r6597\" (UID: \"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:53.101518 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.101487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:53.239180 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.239151    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"]
Jul 27 22:11:53.241184 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:11:53.241144    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c0a458_2792_4a2f_91a6_3e0cf4eb18c0.slice/crio-8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0 WatchSource:0}: Error finding container 8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0: Status 404 returned error can't find the container with id 8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0
Jul 27 22:11:53.243379 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.243364    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:11:53.735297 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:53.735261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597" event={"ID":"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0","Type":"ContainerStarted","Data":"8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0"}
Jul 27 22:11:56.752348 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:56.752310    2560 generic.go:356] "Generic (PLEG): container finished" podID="91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" containerID="b53b964fb992210af0097cefc0abc885a867da686a41749d7e4f6f0cbf4a244a" exitCode=0
Jul 27 22:11:56.752763 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:56.752373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597" event={"ID":"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0","Type":"ContainerDied","Data":"b53b964fb992210af0097cefc0abc885a867da686a41749d7e4f6f0cbf4a244a"}
Jul 27 22:11:57.888422 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:57.888388    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:57.909988 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:57.909956    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca\") pod \"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0\" (UID: \"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0\") "
Jul 27 22:11:57.910297 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:57.910272    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" (UID: "91c0a458-2792-4a2f-91a6-3e0cf4eb18c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:11:58.011646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:58.011555    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:11:58.763234 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:58.763204    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"
Jul 27 22:11:58.763423 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:58.763209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597" event={"ID":"91c0a458-2792-4a2f-91a6-3e0cf4eb18c0","Type":"ContainerDied","Data":"8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0"}
Jul 27 22:11:58.763423 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:11:58.763323    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8743f7c935ba8999de22085556a8c6d93522810ebe076b2bc8ee108736aadba0"
Jul 27 22:12:06.155916 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:06.155881    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:12:07.870163 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:07.870123    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"]
Jul 27 22:12:07.871842 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:07.871817    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66e79c4-r6597"]
Jul 27 22:12:08.915215 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:08.915180    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" path="/var/lib/kubelet/pods/91c0a458-2792-4a2f-91a6-3e0cf4eb18c0/volumes"
Jul 27 22:12:17.660556 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:17.660516    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:12:37.191717 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.191680    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"]
Jul 27 22:12:37.192750 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.192710    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" containerName="revoke-keys"
Jul 27 22:12:37.192750 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.192751    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" containerName="revoke-keys"
Jul 27 22:12:37.192970 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.192956    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="91c0a458-2792-4a2f-91a6-3e0cf4eb18c0" containerName="revoke-keys"
Jul 27 22:12:37.197257 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.197228    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:37.200511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.200483    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:12:37.200640 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.200534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:12:37.204243 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.204220    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"]
Jul 27 22:12:37.287604 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.287560    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz\" (UID: \"f9e356cf-bb76-44bb-99a6-f19183461b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:37.388758 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.388715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz\" (UID: \"f9e356cf-bb76-44bb-99a6-f19183461b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:37.389323 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.389303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz\" (UID: \"f9e356cf-bb76-44bb-99a6-f19183461b6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:37.510530 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.510420    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:37.647754 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.647717    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"]
Jul 27 22:12:37.651082 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:12:37.651050    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e356cf_bb76_44bb_99a6_f19183461b6a.slice/crio-8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93 WatchSource:0}: Error finding container 8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93: Status 404 returned error can't find the container with id 8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93
Jul 27 22:12:37.951786 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:37.951744    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz" event={"ID":"f9e356cf-bb76-44bb-99a6-f19183461b6a","Type":"ContainerStarted","Data":"8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93"}
Jul 27 22:12:38.957678 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:38.957642    2560 generic.go:356] "Generic (PLEG): container finished" podID="f9e356cf-bb76-44bb-99a6-f19183461b6a" containerID="6b8b5e718459c005dc84f39935b89094628cda27716f1fd2df97ac9e67666235" exitCode=0
Jul 27 22:12:38.958076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:38.957730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz" event={"ID":"f9e356cf-bb76-44bb-99a6-f19183461b6a","Type":"ContainerDied","Data":"6b8b5e718459c005dc84f39935b89094628cda27716f1fd2df97ac9e67666235"}
Jul 27 22:12:40.098773 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.098740    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:40.112385 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.112359    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca\") pod \"f9e356cf-bb76-44bb-99a6-f19183461b6a\" (UID: \"f9e356cf-bb76-44bb-99a6-f19183461b6a\") "
Jul 27 22:12:40.112691 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.112669    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9e356cf-bb76-44bb-99a6-f19183461b6a" (UID: "f9e356cf-bb76-44bb-99a6-f19183461b6a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:12:40.213052 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.212966    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9e356cf-bb76-44bb-99a6-f19183461b6a-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:12:40.968580 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.968539    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz" event={"ID":"f9e356cf-bb76-44bb-99a6-f19183461b6a","Type":"ContainerDied","Data":"8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93"}
Jul 27 22:12:40.968580 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.968576    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"
Jul 27 22:12:40.968580 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:40.968588    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e3ae6972e6d585923362ccc8698233580876b09fd17baa87cec30f95f40ef93"
Jul 27 22:12:50.687634 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:50.687597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:12:52.268433 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:52.268397    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"]
Jul 27 22:12:52.270699 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:52.270675    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d2830c6-nbvjz"]
Jul 27 22:12:52.915771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:12:52.915728    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9e356cf-bb76-44bb-99a6-f19183461b6a" path="/var/lib/kubelet/pods/f9e356cf-bb76-44bb-99a6-f19183461b6a/volumes"
Jul 27 22:13:01.855196 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:01.855157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:13:21.170675 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.170646    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"]
Jul 27 22:13:21.171168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.171054    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9e356cf-bb76-44bb-99a6-f19183461b6a" containerName="revoke-keys"
Jul 27 22:13:21.171168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.171066    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9e356cf-bb76-44bb-99a6-f19183461b6a" containerName="revoke-keys"
Jul 27 22:13:21.171168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.171126    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9e356cf-bb76-44bb-99a6-f19183461b6a" containerName="revoke-keys"
Jul 27 22:13:21.174426 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.174401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:21.176917 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.176897    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:13:21.177023 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.176925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:13:21.181646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.181619    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"]
Jul 27 22:13:21.212160 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.212120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b\" (UID: \"dfa2b5b6-0b25-4fa9-9574-1f8430f608be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:21.313718 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.313677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b\" (UID: \"dfa2b5b6-0b25-4fa9-9574-1f8430f608be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:21.314274 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.314253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b\" (UID: \"dfa2b5b6-0b25-4fa9-9574-1f8430f608be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:21.487378 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.487289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:21.619658 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:21.619633    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"]
Jul 27 22:13:21.621400 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:13:21.621372    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa2b5b6_0b25_4fa9_9574_1f8430f608be.slice/crio-65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347 WatchSource:0}: Error finding container 65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347: Status 404 returned error can't find the container with id 65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347
Jul 27 22:13:22.173994 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:22.173911    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b" event={"ID":"dfa2b5b6-0b25-4fa9-9574-1f8430f608be","Type":"ContainerStarted","Data":"4e67a2a597850b3eb5184cc050c91ff86f9939e00f48b099f77246d3ef26898e"}
Jul 27 22:13:22.173994 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:22.173961    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b" event={"ID":"dfa2b5b6-0b25-4fa9-9574-1f8430f608be","Type":"ContainerStarted","Data":"65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347"}
Jul 27 22:13:22.189685 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:22.189638    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b" podStartSLOduration=1.189621894 podStartE2EDuration="1.189621894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:13:22.189061059 +0000 UTC m=+1667.842489701" watchObservedRunningTime="2026-07-27 22:13:22.189621894 +0000 UTC m=+1667.843050556"
Jul 27 22:13:23.180003 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:23.179962    2560 generic.go:356] "Generic (PLEG): container finished" podID="dfa2b5b6-0b25-4fa9-9574-1f8430f608be" containerID="4e67a2a597850b3eb5184cc050c91ff86f9939e00f48b099f77246d3ef26898e" exitCode=0
Jul 27 22:13:23.180437 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:23.180062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b" event={"ID":"dfa2b5b6-0b25-4fa9-9574-1f8430f608be","Type":"ContainerDied","Data":"4e67a2a597850b3eb5184cc050c91ff86f9939e00f48b099f77246d3ef26898e"}
Jul 27 22:13:24.315038 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:24.315014    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:24.444411 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:24.444316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca\") pod \"dfa2b5b6-0b25-4fa9-9574-1f8430f608be\" (UID: \"dfa2b5b6-0b25-4fa9-9574-1f8430f608be\") "
Jul 27 22:13:24.444733 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:24.444708    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfa2b5b6-0b25-4fa9-9574-1f8430f608be" (UID: "dfa2b5b6-0b25-4fa9-9574-1f8430f608be"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:13:24.545325 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:24.545288    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfa2b5b6-0b25-4fa9-9574-1f8430f608be-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:13:25.190806 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:25.190721    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"
Jul 27 22:13:25.190806 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:25.190732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b" event={"ID":"dfa2b5b6-0b25-4fa9-9574-1f8430f608be","Type":"ContainerDied","Data":"65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347"}
Jul 27 22:13:25.190806 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:25.190776    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ece3303e182859695dabe7114915e648afa06119a62ac21c3f02244f371347"
Jul 27 22:13:34.563074 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:34.563041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:13:36.255432 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:36.255391    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"]
Jul 27 22:13:36.259134 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:36.259108    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d4157b7d-ktv2b"]
Jul 27 22:13:36.915808 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:36.915762    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa2b5b6-0b25-4fa9-9574-1f8430f608be" path="/var/lib/kubelet/pods/dfa2b5b6-0b25-4fa9-9574-1f8430f608be/volumes"
Jul 27 22:13:46.160169 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:13:46.160131    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:14:05.403170 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.403093    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"]
Jul 27 22:14:05.403702 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.403684    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa2b5b6-0b25-4fa9-9574-1f8430f608be" containerName="revoke-keys"
Jul 27 22:14:05.403748 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.403705    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa2b5b6-0b25-4fa9-9574-1f8430f608be" containerName="revoke-keys"
Jul 27 22:14:05.403839 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.403828    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa2b5b6-0b25-4fa9-9574-1f8430f608be" containerName="revoke-keys"
Jul 27 22:14:05.406539 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.406521    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:05.408904 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.408884    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:14:05.409025 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.408911    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:14:05.414410 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.414328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"]
Jul 27 22:14:05.541008 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.540970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp\" (UID: \"187a6ecf-c325-46d0-aa1d-2f14be22ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:05.642666 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.642622    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp\" (UID: \"187a6ecf-c325-46d0-aa1d-2f14be22ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:05.643312 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.643291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp\" (UID: \"187a6ecf-c325-46d0-aa1d-2f14be22ac64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:05.718985 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.718895    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:05.855027 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:05.854997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"]
Jul 27 22:14:05.857629 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:14:05.857599    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187a6ecf_c325_46d0_aa1d_2f14be22ac64.slice/crio-c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471 WatchSource:0}: Error finding container c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471: Status 404 returned error can't find the container with id c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471
Jul 27 22:14:06.412767 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:06.412670    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp" event={"ID":"187a6ecf-c325-46d0-aa1d-2f14be22ac64","Type":"ContainerStarted","Data":"6bb3b2e6eaed072b13f0320884c119fb5c126714fe6dba0ef0dc4fe9a52191d2"}
Jul 27 22:14:06.412767 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:06.412728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp" event={"ID":"187a6ecf-c325-46d0-aa1d-2f14be22ac64","Type":"ContainerStarted","Data":"c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471"}
Jul 27 22:14:06.433654 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:06.433603    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp" podStartSLOduration=1.433583604 podStartE2EDuration="1.433583604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:14:06.432372148 +0000 UTC m=+1712.085800789" watchObservedRunningTime="2026-07-27 22:14:06.433583604 +0000 UTC m=+1712.087012247"
Jul 27 22:14:07.425413 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:07.425370    2560 generic.go:356] "Generic (PLEG): container finished" podID="187a6ecf-c325-46d0-aa1d-2f14be22ac64" containerID="6bb3b2e6eaed072b13f0320884c119fb5c126714fe6dba0ef0dc4fe9a52191d2" exitCode=0
Jul 27 22:14:07.425883 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:07.425478    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp" event={"ID":"187a6ecf-c325-46d0-aa1d-2f14be22ac64","Type":"ContainerDied","Data":"6bb3b2e6eaed072b13f0320884c119fb5c126714fe6dba0ef0dc4fe9a52191d2"}
Jul 27 22:14:08.562936 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:08.562911    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:08.671955 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:08.671921    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca\") pod \"187a6ecf-c325-46d0-aa1d-2f14be22ac64\" (UID: \"187a6ecf-c325-46d0-aa1d-2f14be22ac64\") "
Jul 27 22:14:08.672288 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:08.672263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "187a6ecf-c325-46d0-aa1d-2f14be22ac64" (UID: "187a6ecf-c325-46d0-aa1d-2f14be22ac64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:14:08.773181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:08.773076    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/187a6ecf-c325-46d0-aa1d-2f14be22ac64-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:14:09.436309 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:09.436273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp" event={"ID":"187a6ecf-c325-46d0-aa1d-2f14be22ac64","Type":"ContainerDied","Data":"c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471"}
Jul 27 22:14:09.436520 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:09.436323    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c50e0d0f2e1a66beec86b33dbe5525e067a134926007f94af88e717646e32471"
Jul 27 22:14:09.436520 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:09.436288    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"
Jul 27 22:14:18.955380 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:18.955328    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:14:20.475388 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:20.475336    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"]
Jul 27 22:14:20.478611 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:20.478587    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-78fe627e-wtxjp"]
Jul 27 22:14:20.916079 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:14:20.916048    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="187a6ecf-c325-46d0-aa1d-2f14be22ac64" path="/var/lib/kubelet/pods/187a6ecf-c325-46d0-aa1d-2f14be22ac64/volumes"
Jul 27 22:15:47.367772 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:15:47.367728    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:15:57.457798 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:15:57.457760    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:16:08.654303 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:08.654260    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:16:14.745117 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.745078    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"]
Jul 27 22:16:14.745615 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.745469    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="187a6ecf-c325-46d0-aa1d-2f14be22ac64" containerName="revoke-keys"
Jul 27 22:16:14.745615 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.745483    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="187a6ecf-c325-46d0-aa1d-2f14be22ac64" containerName="revoke-keys"
Jul 27 22:16:14.745615 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.745563    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="187a6ecf-c325-46d0-aa1d-2f14be22ac64" containerName="revoke-keys"
Jul 27 22:16:14.748627 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.748611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:14.751811 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.751788    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:16:14.752255 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.752241    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:16:14.760316 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.760290    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"]
Jul 27 22:16:14.856166 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.856128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2671d368-xqnzq\" (UID: \"81f3329e-a0f1-4236-98d2-7ca5bc74056b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:14.957515 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.957480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2671d368-xqnzq\" (UID: \"81f3329e-a0f1-4236-98d2-7ca5bc74056b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:14.958089 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:14.958067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2671d368-xqnzq\" (UID: \"81f3329e-a0f1-4236-98d2-7ca5bc74056b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:15.058987 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:15.058953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:15.200204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:15.200177    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"]
Jul 27 22:16:15.201359 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:16:15.201333    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f3329e_a0f1_4236_98d2_7ca5bc74056b.slice/crio-75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580 WatchSource:0}: Error finding container 75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580: Status 404 returned error can't find the container with id 75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580
Jul 27 22:16:16.081746 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:16.081706    2560 generic.go:356] "Generic (PLEG): container finished" podID="81f3329e-a0f1-4236-98d2-7ca5bc74056b" containerID="87ee1e7c6b331830e48f1090a59edec893bf454bee584bf621638708baada76b" exitCode=0
Jul 27 22:16:16.082161 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:16.081791    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq" event={"ID":"81f3329e-a0f1-4236-98d2-7ca5bc74056b","Type":"ContainerDied","Data":"87ee1e7c6b331830e48f1090a59edec893bf454bee584bf621638708baada76b"}
Jul 27 22:16:16.082161 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:16.081866    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq" event={"ID":"81f3329e-a0f1-4236-98d2-7ca5bc74056b","Type":"ContainerStarted","Data":"75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580"}
Jul 27 22:16:17.220983 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:17.220955    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:17.384545 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:17.384422    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca\") pod \"81f3329e-a0f1-4236-98d2-7ca5bc74056b\" (UID: \"81f3329e-a0f1-4236-98d2-7ca5bc74056b\") "
Jul 27 22:16:17.384838 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:17.384813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81f3329e-a0f1-4236-98d2-7ca5bc74056b" (UID: "81f3329e-a0f1-4236-98d2-7ca5bc74056b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:16:17.485897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:17.485854    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81f3329e-a0f1-4236-98d2-7ca5bc74056b-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:16:18.092732 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:18.092697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"
Jul 27 22:16:18.092915 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:18.092731    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq" event={"ID":"81f3329e-a0f1-4236-98d2-7ca5bc74056b","Type":"ContainerDied","Data":"75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580"}
Jul 27 22:16:18.092915 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:18.092777    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75bb0b4dd53e405c5f37e47310b450cc337982b51560a04de0ea5bcdfd446580"
Jul 27 22:16:28.066949 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:28.066909    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:16:29.827527 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:29.827487    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"]
Jul 27 22:16:29.830536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:29.830504    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2671d368-xqnzq"]
Jul 27 22:16:30.915499 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:16:30.915442    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f3329e-a0f1-4236-98d2-7ca5bc74056b" path="/var/lib/kubelet/pods/81f3329e-a0f1-4236-98d2-7ca5bc74056b/volumes"
Jul 27 22:17:05.758717 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:05.758635    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:17:27.946972 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.946937    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:17:27.947396 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.947378    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f3329e-a0f1-4236-98d2-7ca5bc74056b" containerName="revoke-keys"
Jul 27 22:17:27.947484 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.947400    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f3329e-a0f1-4236-98d2-7ca5bc74056b" containerName="revoke-keys"
Jul 27 22:17:27.947530 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.947485    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f3329e-a0f1-4236-98d2-7ca5bc74056b" containerName="revoke-keys"
Jul 27 22:17:27.950620 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.950603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:27.954381 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.954356    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5d6c9eb6\"/\"default-dockercfg-fj5q5\""
Jul 27 22:17:27.954526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.954357    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5d6c9eb6\"/\"kube-root-ca.crt\""
Jul 27 22:17:27.954526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.954357    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5d6c9eb6\"/\"e2e-lifecycle-model-5d6c9eb6-kserve-self-signed-certs\""
Jul 27 22:17:27.954526 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.954368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5d6c9eb6\"/\"openshift-service-ca.crt\""
Jul 27 22:17:27.961481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:27.961443    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:17:28.060272 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.060471 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.060471 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2v9\" (UniqueName: \"kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.060471 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.060471 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060371    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.060471 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.060437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.161777 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.161777 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2v9\" (UniqueName: \"kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161838    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161864    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.161921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162265 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.162205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162318 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.162264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.162318 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.162274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.164277 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.164257    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.164573 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.164554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.172154 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.172124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2v9\" (UniqueName: \"kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9\") pod \"e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") " pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.261371 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.261277    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:28.405892 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.405866    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:17:28.408513 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:17:28.408477    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35a9b42a_68b0_450d_b35e_d63d3850def2.slice/crio-6f3136e58ec420b546643e054bb323d4518f399a6103514cb19e92dc3eb01eea WatchSource:0}: Error finding container 6f3136e58ec420b546643e054bb323d4518f399a6103514cb19e92dc3eb01eea: Status 404 returned error can't find the container with id 6f3136e58ec420b546643e054bb323d4518f399a6103514cb19e92dc3eb01eea
Jul 27 22:17:28.444674 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:28.444642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" event={"ID":"35a9b42a-68b0-450d-b35e-d63d3850def2","Type":"ContainerStarted","Data":"6f3136e58ec420b546643e054bb323d4518f399a6103514cb19e92dc3eb01eea"}
Jul 27 22:17:29.451302 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:29.451265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" event={"ID":"35a9b42a-68b0-450d-b35e-d63d3850def2","Type":"ContainerStarted","Data":"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"}
Jul 27 22:17:29.469426 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:29.469374    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" podStartSLOduration=2.469359198 podStartE2EDuration="2.469359198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:17:29.467952787 +0000 UTC m=+1915.121381429" watchObservedRunningTime="2026-07-27 22:17:29.469359198 +0000 UTC m=+1915.122787839"
Jul 27 22:17:30.854839 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:30.854805    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:17:38.261733 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:38.261690    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:38.262345 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:38.261756    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:38.274043 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:38.274003    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:38.508559 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:38.508530    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:17:59.474303 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:17:59.474269    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:18:16.649680 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:16.649639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:18:28.453356 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:28.453276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:18:32.319075 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:32.319034    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:18:34.972981 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:34.972947    2560 scope.go:117] "RemoveContainer" containerID="b53b964fb992210af0097cefc0abc885a867da686a41749d7e4f6f0cbf4a244a"
Jul 27 22:18:47.339120 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.339079    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" podUID="35a9b42a-68b0-450d-b35e-d63d3850def2" containerName="main" containerID="cri-o://3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da" gracePeriod=45
Jul 27 22:18:47.594404 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.594349    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:18:47.726340 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726308    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726340 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726373    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp2v9\" (UniqueName: \"kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726472    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726517    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726580    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache\") pod \"35a9b42a-68b0-450d-b35e-d63d3850def2\" (UID: \"35a9b42a-68b0-450d-b35e-d63d3850def2\") "
Jul 27 22:18:47.726622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:18:47.726896 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726763    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home" (OuterVolumeSpecName: "home") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:18:47.726896 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726809    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache" (OuterVolumeSpecName: "model-cache") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:18:47.726975 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726914    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.726975 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726932    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.726975 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.726945    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.728791 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.728768    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:18:47.728879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.728771    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm" (OuterVolumeSpecName: "dshm") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:18:47.728879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.728863    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9" (OuterVolumeSpecName: "kube-api-access-xp2v9") pod "35a9b42a-68b0-450d-b35e-d63d3850def2" (UID: "35a9b42a-68b0-450d-b35e-d63d3850def2"). InnerVolumeSpecName "kube-api-access-xp2v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:18:47.827368 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.827328    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a9b42a-68b0-450d-b35e-d63d3850def2-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.827368 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.827364    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2v9\" (UniqueName: \"kubernetes.io/projected/35a9b42a-68b0-450d-b35e-d63d3850def2-kube-api-access-xp2v9\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.827575 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.827378    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a9b42a-68b0-450d-b35e-d63d3850def2-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:18:47.854184 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.854110    2560 generic.go:356] "Generic (PLEG): container finished" podID="35a9b42a-68b0-450d-b35e-d63d3850def2" containerID="3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da" exitCode=0
Jul 27 22:18:47.854184 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.854152    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"
Jul 27 22:18:47.854184 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.854179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" event={"ID":"35a9b42a-68b0-450d-b35e-d63d3850def2","Type":"ContainerDied","Data":"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"}
Jul 27 22:18:47.854386 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.854213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg" event={"ID":"35a9b42a-68b0-450d-b35e-d63d3850def2","Type":"ContainerDied","Data":"6f3136e58ec420b546643e054bb323d4518f399a6103514cb19e92dc3eb01eea"}
Jul 27 22:18:47.854386 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.854230    2560 scope.go:117] "RemoveContainer" containerID="3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"
Jul 27 22:18:47.863829 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.863812    2560 scope.go:117] "RemoveContainer" containerID="3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"
Jul 27 22:18:47.864069 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:18:47.864051    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da\": container with ID starting with 3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da not found: ID does not exist" containerID="3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"
Jul 27 22:18:47.864134 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.864080    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da"} err="failed to get container status \"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da\": rpc error: code = NotFound desc = could not find container \"3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da\": container with ID starting with 3ba00c82b1a021c73f72e9dbc98ebfde7e4e8ca2fd93427c4c1cbf99c9c2b5da not found: ID does not exist"
Jul 27 22:18:47.876645 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.876619    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:18:47.880162 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:47.880138    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5d6c9eb6/e2e-lifecycle-model-5d6c9eb6-kserve-db6fd8db7-j7vkg"]
Jul 27 22:18:48.915788 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:18:48.915755    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a9b42a-68b0-450d-b35e-d63d3850def2" path="/var/lib/kubelet/pods/35a9b42a-68b0-450d-b35e-d63d3850def2/volumes"
Jul 27 22:19:21.656823 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:19:21.656790    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:19:30.853702 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:19:30.853667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:19:34.995387 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:19:34.995354    2560 scope.go:117] "RemoveContainer" containerID="6b8b5e718459c005dc84f39935b89094628cda27716f1fd2df97ac9e67666235"
Jul 27 22:19:35.004078 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:19:35.003967    2560 scope.go:117] "RemoveContainer" containerID="4e67a2a597850b3eb5184cc050c91ff86f9939e00f48b099f77246d3ef26898e"
Jul 27 22:20:18.948282 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:18.948171    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:20:30.060636 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:30.060602    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:20:33.045208 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.045172    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:20:33.045760 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.045740    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a9b42a-68b0-450d-b35e-d63d3850def2" containerName="main"
Jul 27 22:20:33.045840 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.045763    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a9b42a-68b0-450d-b35e-d63d3850def2" containerName="main"
Jul 27 22:20:33.045897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.045875    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a9b42a-68b0-450d-b35e-d63d3850def2" containerName="main"
Jul 27 22:20:33.049003 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.048982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.052296 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.052275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a6dea\"/\"test-model-7a6dea-kserve-self-signed-certs\""
Jul 27 22:20:33.053344 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.053320    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a6dea\"/\"kube-root-ca.crt\""
Jul 27 22:20:33.053470 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.053345    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a6dea\"/\"openshift-service-ca.crt\""
Jul 27 22:20:33.053470 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.053364    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a6dea\"/\"default-dockercfg-cp46n\""
Jul 27 22:20:33.062080 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.062054    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:20:33.098873 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.098844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.098984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.098894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.098984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.098924    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mvvp\" (UniqueName: \"kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.098984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.098951    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.098984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.098972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.099141 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.099033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200031 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.199997    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200145 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200196 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200196 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mvvp\" (UniqueName: \"kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200285 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200285 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200248    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200504 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200600 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.200600 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.200590    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.202340 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.202321    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.202899 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.202882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.212861 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.212843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mvvp\" (UniqueName: \"kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp\") pod \"test-model-7a6dea-kserve-649d7dd669-hgcs7\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") " pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.357914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.357890    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:33.493693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:33.493669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:20:33.496925 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:20:33.496895    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67a49b6_221d_4b82_949d_7bb1582b27c8.slice/crio-7587db2862f808553aba7e9f99e4936c7f9f594a3dcd02f1e055f00f892381c7 WatchSource:0}: Error finding container 7587db2862f808553aba7e9f99e4936c7f9f594a3dcd02f1e055f00f892381c7: Status 404 returned error can't find the container with id 7587db2862f808553aba7e9f99e4936c7f9f594a3dcd02f1e055f00f892381c7
Jul 27 22:20:34.374505 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:34.374435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" event={"ID":"f67a49b6-221d-4b82-949d-7bb1582b27c8","Type":"ContainerStarted","Data":"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"}
Jul 27 22:20:34.374923 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:34.374515    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" event={"ID":"f67a49b6-221d-4b82-949d-7bb1582b27c8","Type":"ContainerStarted","Data":"7587db2862f808553aba7e9f99e4936c7f9f594a3dcd02f1e055f00f892381c7"}
Jul 27 22:20:34.395061 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:34.395005    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" podStartSLOduration=1.394984558 podStartE2EDuration="1.394984558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:20:34.39366972 +0000 UTC m=+2100.047098362" watchObservedRunningTime="2026-07-27 22:20:34.394984558 +0000 UTC m=+2100.048413200"
Jul 27 22:20:35.031574 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:35.031543    2560 scope.go:117] "RemoveContainer" containerID="6bb3b2e6eaed072b13f0320884c119fb5c126714fe6dba0ef0dc4fe9a52191d2"
Jul 27 22:20:39.169594 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:39.169553    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:20:43.358536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:43.358500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:43.358536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:43.358545    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:43.371083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:43.371060    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:43.428637 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:43.428613    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:20:50.082036 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:20:50.081997    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:21:16.964417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.964388    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:21:16.967943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.967922    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:16.975856 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.975833    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-00544d\"/\"openshift-service-ca.crt\""
Jul 27 22:21:16.975960 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.975923    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-00544d\"/\"default-dockercfg-qrnfp\""
Jul 27 22:21:16.976181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.976165    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-00544d\"/\"test-model-00544d-kserve-self-signed-certs\""
Jul 27 22:21:16.976656 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.976641    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-00544d\"/\"kube-root-ca.crt\""
Jul 27 22:21:16.984519 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:16.984499    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:21:17.002859 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.002825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cssgg\" (UniqueName: \"kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.002962 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.002869    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.002962 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.002938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.002962 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.002955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.003075 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.002991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.003075 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.003014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104242 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104242 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104485 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104485 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104485 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cssgg\" (UniqueName: \"kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104662 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104726 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104659    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104726 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.104865 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.104841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.106774 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.106749    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.107029 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.107013    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.117050 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.117032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cssgg\" (UniqueName: \"kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg\") pod \"test-model-00544d-kserve-ddb748555-2x4pd\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") " pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.277670 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.277589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:17.418987 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.418964    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:21:17.421847 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:21:17.421814    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a8b5385_c20d_477e_a112_ffc79e527672.slice/crio-78a6994a9034a8b98b588cffd30d7b16c44d8fcecbcf7a4e47f9e44c85bc183d WatchSource:0}: Error finding container 78a6994a9034a8b98b588cffd30d7b16c44d8fcecbcf7a4e47f9e44c85bc183d: Status 404 returned error can't find the container with id 78a6994a9034a8b98b588cffd30d7b16c44d8fcecbcf7a4e47f9e44c85bc183d
Jul 27 22:21:17.595520 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.595471    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" event={"ID":"6a8b5385-c20d-477e-a112-ffc79e527672","Type":"ContainerStarted","Data":"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"}
Jul 27 22:21:17.595520 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.595522    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" event={"ID":"6a8b5385-c20d-477e-a112-ffc79e527672","Type":"ContainerStarted","Data":"78a6994a9034a8b98b588cffd30d7b16c44d8fcecbcf7a4e47f9e44c85bc183d"}
Jul 27 22:21:17.615257 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:17.615202    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" podStartSLOduration=1.6151861589999998 podStartE2EDuration="1.615186159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:21:17.613510036 +0000 UTC m=+2143.266938702" watchObservedRunningTime="2026-07-27 22:21:17.615186159 +0000 UTC m=+2143.268614800"
Jul 27 22:21:22.559792 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:22.559755    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:21:27.278698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:27.278590    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:27.278698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:27.278669    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:27.291473 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:27.291431    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:27.656943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:27.656908    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:21:56.880261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:56.880220    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:21:57.814652 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.814616    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:21:57.962884 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.962851    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"]
Jul 27 22:21:57.966542 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.966525    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:21:57.969236 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.969218    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:21:57.969356 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.969337    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:21:57.979986 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:57.979964    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"]
Jul 27 22:21:58.000115 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.000086    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn\" (UID: \"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:21:58.101321 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.101240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn\" (UID: \"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:21:58.101985 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.101961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn\" (UID: \"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:21:58.276901 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.276867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:21:58.415958 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.415935    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"]
Jul 27 22:21:58.418731 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:21:58.418703    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bef6b76_e40d_4aa6_bc79_6f5cd84b231c.slice/crio-02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980 WatchSource:0}: Error finding container 02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980: Status 404 returned error can't find the container with id 02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980
Jul 27 22:21:58.797414 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.797382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn" event={"ID":"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c","Type":"ContainerStarted","Data":"7589138cabe280b8e4cae3221df3306d4b9f3cb8c780b99477c03c5d972e02d8"}
Jul 27 22:21:58.797609 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.797444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn" event={"ID":"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c","Type":"ContainerStarted","Data":"02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980"}
Jul 27 22:21:58.817599 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:58.817537    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn" podStartSLOduration=1.817519211 podStartE2EDuration="1.817519211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:21:58.816309304 +0000 UTC m=+2184.469737947" watchObservedRunningTime="2026-07-27 22:21:58.817519211 +0000 UTC m=+2184.470947855"
Jul 27 22:21:59.804709 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:59.804678    2560 generic.go:356] "Generic (PLEG): container finished" podID="3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" containerID="7589138cabe280b8e4cae3221df3306d4b9f3cb8c780b99477c03c5d972e02d8" exitCode=0
Jul 27 22:21:59.805090 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:21:59.804759    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn" event={"ID":"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c","Type":"ContainerDied","Data":"7589138cabe280b8e4cae3221df3306d4b9f3cb8c780b99477c03c5d972e02d8"}
Jul 27 22:22:00.945058 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:00.945036    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:22:01.029359 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.029327    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca\") pod \"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c\" (UID: \"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c\") "
Jul 27 22:22:01.029726 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.029701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" (UID: "3bef6b76-e40d-4aa6-bc79-6f5cd84b231c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:01.130057 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.129984    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:01.815960 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.815923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn" event={"ID":"3bef6b76-e40d-4aa6-bc79-6f5cd84b231c","Type":"ContainerDied","Data":"02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980"}
Jul 27 22:22:01.816123 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.815978    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b846b793046cdccde7f35a1e8f97d21bad85be172fc21ae98152a804060980"
Jul 27 22:22:01.816123 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:01.815946    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"
Jul 27 22:22:02.174905 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:02.174822    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:22:11.899728 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:11.899681    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" podUID="f67a49b6-221d-4b82-949d-7bb1582b27c8" containerName="main" containerID="cri-o://32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6" gracePeriod=45
Jul 27 22:22:12.154085 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.154026    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:22:12.235606 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.235805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235641    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.235805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235666    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.235805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235734    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mvvp\" (UniqueName: \"kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.235805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.235805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235794    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs\") pod \"f67a49b6-221d-4b82-949d-7bb1582b27c8\" (UID: \"f67a49b6-221d-4b82-949d-7bb1582b27c8\") "
Jul 27 22:22:12.236057 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235902    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:12.236057 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.235914    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home" (OuterVolumeSpecName: "home") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:12.236153 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.236085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:12.236153 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.236145    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.236234 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.236164    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.238042 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.238020    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm" (OuterVolumeSpecName: "dshm") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:12.238191 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.238167    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:22:12.238248 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.238190    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp" (OuterVolumeSpecName: "kube-api-access-6mvvp") pod "f67a49b6-221d-4b82-949d-7bb1582b27c8" (UID: "f67a49b6-221d-4b82-949d-7bb1582b27c8"). InnerVolumeSpecName "kube-api-access-6mvvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:12.336973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.336935    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.336973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.336965    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mvvp\" (UniqueName: \"kubernetes.io/projected/f67a49b6-221d-4b82-949d-7bb1582b27c8-kube-api-access-6mvvp\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.336973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.336977    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f67a49b6-221d-4b82-949d-7bb1582b27c8-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.337202 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.336985    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f67a49b6-221d-4b82-949d-7bb1582b27c8-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:12.834132 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.834088    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" podUID="6a8b5385-c20d-477e-a112-ffc79e527672" containerName="main" containerID="cri-o://8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841" gracePeriod=45
Jul 27 22:22:12.875024 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.874993    2560 generic.go:356] "Generic (PLEG): container finished" podID="f67a49b6-221d-4b82-949d-7bb1582b27c8" containerID="32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6" exitCode=0
Jul 27 22:22:12.875166 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.875035    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"
Jul 27 22:22:12.875166 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.875076    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" event={"ID":"f67a49b6-221d-4b82-949d-7bb1582b27c8","Type":"ContainerDied","Data":"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"}
Jul 27 22:22:12.875166 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.875117    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7" event={"ID":"f67a49b6-221d-4b82-949d-7bb1582b27c8","Type":"ContainerDied","Data":"7587db2862f808553aba7e9f99e4936c7f9f594a3dcd02f1e055f00f892381c7"}
Jul 27 22:22:12.875166 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.875135    2560 scope.go:117] "RemoveContainer" containerID="32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"
Jul 27 22:22:12.907085 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.907057    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:22:12.917174 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.917151    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7a6dea/test-model-7a6dea-kserve-649d7dd669-hgcs7"]
Jul 27 22:22:12.944924 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.944828    2560 scope.go:117] "RemoveContainer" containerID="32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"
Jul 27 22:22:12.945164 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:22:12.945138    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6\": container with ID starting with 32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6 not found: ID does not exist" containerID="32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"
Jul 27 22:22:12.945260 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:12.945166    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6"} err="failed to get container status \"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6\": rpc error: code = NotFound desc = could not find container \"32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6\": container with ID starting with 32994ad65e9ddbe87d3e306f2df9bae34b2cabe96fd219ea502cdd952df89fe6 not found: ID does not exist"
Jul 27 22:22:13.089930 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.089866    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:22:13.238900 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.238868    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"]
Jul 27 22:22:13.245073 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.245048    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7a6dea-zfzvn"]
Jul 27 22:22:13.246613 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246594    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246659 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246625    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cssgg\" (UniqueName: \"kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246659 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246650    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246727 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246768 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246742    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246906 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246863    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir\") pod \"6a8b5385-c20d-477e-a112-ffc79e527672\" (UID: \"6a8b5385-c20d-477e-a112-ffc79e527672\") "
Jul 27 22:22:13.246992 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246981    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home" (OuterVolumeSpecName: "home") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:13.247054 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.246998    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:13.247110 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.247080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:13.247409 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.247393    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.247522 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.247414    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.247522 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.247428    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.249013 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.248994    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg" (OuterVolumeSpecName: "kube-api-access-cssgg") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "kube-api-access-cssgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:22:13.249076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.249007    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:22:13.249076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.249049    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm" (OuterVolumeSpecName: "dshm") pod "6a8b5385-c20d-477e-a112-ffc79e527672" (UID: "6a8b5385-c20d-477e-a112-ffc79e527672"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:22:13.348114 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.348028    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a8b5385-c20d-477e-a112-ffc79e527672-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.348114 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.348056    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cssgg\" (UniqueName: \"kubernetes.io/projected/6a8b5385-c20d-477e-a112-ffc79e527672-kube-api-access-cssgg\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.348114 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.348065    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a8b5385-c20d-477e-a112-ffc79e527672-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:13.882983 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.882949    2560 generic.go:356] "Generic (PLEG): container finished" podID="6a8b5385-c20d-477e-a112-ffc79e527672" containerID="8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841" exitCode=0
Jul 27 22:22:13.883168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.882992    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"
Jul 27 22:22:13.883168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.883018    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" event={"ID":"6a8b5385-c20d-477e-a112-ffc79e527672","Type":"ContainerDied","Data":"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"}
Jul 27 22:22:13.883168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.883048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd" event={"ID":"6a8b5385-c20d-477e-a112-ffc79e527672","Type":"ContainerDied","Data":"78a6994a9034a8b98b588cffd30d7b16c44d8fcecbcf7a4e47f9e44c85bc183d"}
Jul 27 22:22:13.883168 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.883064    2560 scope.go:117] "RemoveContainer" containerID="8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"
Jul 27 22:22:13.894437 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.894417    2560 scope.go:117] "RemoveContainer" containerID="8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"
Jul 27 22:22:13.894746 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:22:13.894726    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841\": container with ID starting with 8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841 not found: ID does not exist" containerID="8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"
Jul 27 22:22:13.894811 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.894754    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841"} err="failed to get container status \"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841\": rpc error: code = NotFound desc = could not find container \"8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841\": container with ID starting with 8de1dae289cfe8d82d5c806683efcf85708bf75bee37b498e3e9a1e0bb70f841 not found: ID does not exist"
Jul 27 22:22:13.913842 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.913814    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:22:13.921320 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:13.921290    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-00544d/test-model-00544d-kserve-ddb748555-2x4pd"]
Jul 27 22:22:14.916281 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:14.916236    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" path="/var/lib/kubelet/pods/3bef6b76-e40d-4aa6-bc79-6f5cd84b231c/volumes"
Jul 27 22:22:14.916824 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:14.916803    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a8b5385-c20d-477e-a112-ffc79e527672" path="/var/lib/kubelet/pods/6a8b5385-c20d-477e-a112-ffc79e527672/volumes"
Jul 27 22:22:14.917305 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:14.917283    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f67a49b6-221d-4b82-949d-7bb1582b27c8" path="/var/lib/kubelet/pods/f67a49b6-221d-4b82-949d-7bb1582b27c8/volumes"
Jul 27 22:22:17.424985 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:17.424946    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:22:20.198291 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198257    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"]
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198650    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f67a49b6-221d-4b82-949d-7bb1582b27c8" containerName="main"
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198661    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f67a49b6-221d-4b82-949d-7bb1582b27c8" containerName="main"
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198675    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" containerName="revoke-keys"
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198680    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" containerName="revoke-keys"
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198689    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8b5385-c20d-477e-a112-ffc79e527672" containerName="main"
Jul 27 22:22:20.198697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198696    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8b5385-c20d-477e-a112-ffc79e527672" containerName="main"
Jul 27 22:22:20.198886 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198745    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f67a49b6-221d-4b82-949d-7bb1582b27c8" containerName="main"
Jul 27 22:22:20.198886 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198753    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bef6b76-e40d-4aa6-bc79-6f5cd84b231c" containerName="revoke-keys"
Jul 27 22:22:20.198886 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.198763    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8b5385-c20d-477e-a112-ffc79e527672" containerName="main"
Jul 27 22:22:20.203016 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.202999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:20.205805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.205782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:22:20.205805 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.205803    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:22:20.210492 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.210442    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"]
Jul 27 22:22:20.316589 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.316556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6\" (UID: \"3cfe7d48-e2ef-4415-994c-d7216420a64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:20.417331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.417295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6\" (UID: \"3cfe7d48-e2ef-4415-994c-d7216420a64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:20.417946 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.417922    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6\" (UID: \"3cfe7d48-e2ef-4415-994c-d7216420a64a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:20.515238 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.515156    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:20.655715 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.655682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"]
Jul 27 22:22:20.659092 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:22:20.659066    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfe7d48_e2ef_4415_994c_d7216420a64a.slice/crio-daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e WatchSource:0}: Error finding container daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e: Status 404 returned error can't find the container with id daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e
Jul 27 22:22:20.924331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:20.924301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6" event={"ID":"3cfe7d48-e2ef-4415-994c-d7216420a64a","Type":"ContainerStarted","Data":"daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e"}
Jul 27 22:22:21.930692 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:21.930658    2560 generic.go:356] "Generic (PLEG): container finished" podID="3cfe7d48-e2ef-4415-994c-d7216420a64a" containerID="9870e63c2212d7e66222ad1715be259d5f30144c894a6d8aa1e899004f153312" exitCode=0
Jul 27 22:22:21.931072 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:21.930752    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6" event={"ID":"3cfe7d48-e2ef-4415-994c-d7216420a64a","Type":"ContainerDied","Data":"9870e63c2212d7e66222ad1715be259d5f30144c894a6d8aa1e899004f153312"}
Jul 27 22:22:23.067275 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.067244    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:23.242922 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.242827    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca\") pod \"3cfe7d48-e2ef-4415-994c-d7216420a64a\" (UID: \"3cfe7d48-e2ef-4415-994c-d7216420a64a\") "
Jul 27 22:22:23.243219 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.243191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cfe7d48-e2ef-4415-994c-d7216420a64a" (UID: "3cfe7d48-e2ef-4415-994c-d7216420a64a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:22:23.343543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.343505    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cfe7d48-e2ef-4415-994c-d7216420a64a-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:22:23.940917 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.940883    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6" event={"ID":"3cfe7d48-e2ef-4415-994c-d7216420a64a","Type":"ContainerDied","Data":"daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e"}
Jul 27 22:22:23.940917 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.940911    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"
Jul 27 22:22:23.941143 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:23.940929    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa432d03044c59e701dd5f17f5c51432c331743d31d85e74865786a9c8f7b8e"
Jul 27 22:22:33.366633 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:33.366597    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:22:35.060785 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:35.060751    2560 scope.go:117] "RemoveContainer" containerID="87ee1e7c6b331830e48f1090a59edec893bf454bee584bf621638708baada76b"
Jul 27 22:22:35.322944 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:35.322857    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"]
Jul 27 22:22:35.330323 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:35.330293    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-00544d-hm7f6"]
Jul 27 22:22:36.914992 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:36.914956    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfe7d48-e2ef-4415-994c-d7216420a64a" path="/var/lib/kubelet/pods/3cfe7d48-e2ef-4415-994c-d7216420a64a/volumes"
Jul 27 22:22:52.080012 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:22:52.079978    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:23:00.867894 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:00.867799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:23:08.174550 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.174518    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"]
Jul 27 22:23:08.175025 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.174895    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfe7d48-e2ef-4415-994c-d7216420a64a" containerName="revoke-keys"
Jul 27 22:23:08.175025 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.174907    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfe7d48-e2ef-4415-994c-d7216420a64a" containerName="revoke-keys"
Jul 27 22:23:08.175025 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.174977    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfe7d48-e2ef-4415-994c-d7216420a64a" containerName="revoke-keys"
Jul 27 22:23:08.177736 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.177718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:08.181000 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.180974    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:23:08.181123 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.181080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:23:08.190365 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.190339    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"]
Jul 27 22:23:08.264034 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.264000    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721ee8-xk427\" (UID: \"96cf7934-5127-45b5-a249-6ece51fdfad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:08.364884 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.364852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721ee8-xk427\" (UID: \"96cf7934-5127-45b5-a249-6ece51fdfad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:08.365498 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.365478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-721ee8-xk427\" (UID: \"96cf7934-5127-45b5-a249-6ece51fdfad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:08.487668 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.487584    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:08.624770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:08.624747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"]
Jul 27 22:23:08.626665 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:23:08.626641    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cf7934_5127_45b5_a249_6ece51fdfad7.slice/crio-1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53 WatchSource:0}: Error finding container 1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53: Status 404 returned error can't find the container with id 1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53
Jul 27 22:23:09.171278 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:09.171185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427" event={"ID":"96cf7934-5127-45b5-a249-6ece51fdfad7","Type":"ContainerStarted","Data":"b0676443b4299c65a7bc0830cd69164750b358a7469c05d3415da0873dc3cb30"}
Jul 27 22:23:09.171278 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:09.171246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427" event={"ID":"96cf7934-5127-45b5-a249-6ece51fdfad7","Type":"ContainerStarted","Data":"1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53"}
Jul 27 22:23:09.190474 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:09.190402    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427" podStartSLOduration=1.190388343 podStartE2EDuration="1.190388343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:23:09.188711754 +0000 UTC m=+2254.842140396" watchObservedRunningTime="2026-07-27 22:23:09.190388343 +0000 UTC m=+2254.843817028"
Jul 27 22:23:10.177592 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:10.177558    2560 generic.go:356] "Generic (PLEG): container finished" podID="96cf7934-5127-45b5-a249-6ece51fdfad7" containerID="b0676443b4299c65a7bc0830cd69164750b358a7469c05d3415da0873dc3cb30" exitCode=0
Jul 27 22:23:10.177772 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:10.177648    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427" event={"ID":"96cf7934-5127-45b5-a249-6ece51fdfad7","Type":"ContainerDied","Data":"b0676443b4299c65a7bc0830cd69164750b358a7469c05d3415da0873dc3cb30"}
Jul 27 22:23:11.313536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:11.313507    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:11.391350 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:11.391315    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca\") pod \"96cf7934-5127-45b5-a249-6ece51fdfad7\" (UID: \"96cf7934-5127-45b5-a249-6ece51fdfad7\") "
Jul 27 22:23:11.391690 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:11.391665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96cf7934-5127-45b5-a249-6ece51fdfad7" (UID: "96cf7934-5127-45b5-a249-6ece51fdfad7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:11.492341 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:11.492267    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96cf7934-5127-45b5-a249-6ece51fdfad7-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:23:12.188790 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:12.188762    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"
Jul 27 22:23:12.188978 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:12.188758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427" event={"ID":"96cf7934-5127-45b5-a249-6ece51fdfad7","Type":"ContainerDied","Data":"1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53"}
Jul 27 22:23:12.188978 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:12.188895    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ea02f2108803fda5886250d696f0a299d2426e2fde84f382f24d38031954a53"
Jul 27 22:23:26.656678 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:26.656640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:23:28.258823 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:28.258768    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"]
Jul 27 22:23:28.260999 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:28.260969    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-721ee8-xk427"]
Jul 27 22:23:28.915827 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:28.915795    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cf7934-5127-45b5-a249-6ece51fdfad7" path="/var/lib/kubelet/pods/96cf7934-5127-45b5-a249-6ece51fdfad7/volumes"
Jul 27 22:23:36.130209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.130176    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"]
Jul 27 22:23:36.130606 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.130584    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cf7934-5127-45b5-a249-6ece51fdfad7" containerName="revoke-keys"
Jul 27 22:23:36.130606 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.130597    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cf7934-5127-45b5-a249-6ece51fdfad7" containerName="revoke-keys"
Jul 27 22:23:36.130699 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.130688    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cf7934-5127-45b5-a249-6ece51fdfad7" containerName="revoke-keys"
Jul 27 22:23:36.134761 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.134743    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:36.137414 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.137393    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:23:36.137521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.137393    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:23:36.142230 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.141995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"]
Jul 27 22:23:36.222741 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.222708    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg\" (UID: \"8ac9459b-0707-4d97-8405-3043a1a59fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:36.324180 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.324151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg\" (UID: \"8ac9459b-0707-4d97-8405-3043a1a59fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:36.324764 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.324745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg\" (UID: \"8ac9459b-0707-4d97-8405-3043a1a59fc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:36.445877 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.445806    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:36.578846 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:36.578702    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"]
Jul 27 22:23:36.580991 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:23:36.580969    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac9459b_0707_4d97_8405_3043a1a59fc8.slice/crio-c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9 WatchSource:0}: Error finding container c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9: Status 404 returned error can't find the container with id c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9
Jul 27 22:23:37.314614 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:37.314580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg" event={"ID":"8ac9459b-0707-4d97-8405-3043a1a59fc8","Type":"ContainerStarted","Data":"911eb05950bfbd5fecb1905ff5fc377c7167a48d44a5335b6f07b9623da8bd1d"}
Jul 27 22:23:37.315030 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:37.314630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg" event={"ID":"8ac9459b-0707-4d97-8405-3043a1a59fc8","Type":"ContainerStarted","Data":"c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9"}
Jul 27 22:23:37.332022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:37.331976    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg" podStartSLOduration=1.331962302 podStartE2EDuration="1.331962302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:23:37.330746699 +0000 UTC m=+2282.984175339" watchObservedRunningTime="2026-07-27 22:23:37.331962302 +0000 UTC m=+2282.985390954"
Jul 27 22:23:38.320490 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:38.320432    2560 generic.go:356] "Generic (PLEG): container finished" podID="8ac9459b-0707-4d97-8405-3043a1a59fc8" containerID="911eb05950bfbd5fecb1905ff5fc377c7167a48d44a5335b6f07b9623da8bd1d" exitCode=0
Jul 27 22:23:38.320946 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:38.320505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg" event={"ID":"8ac9459b-0707-4d97-8405-3043a1a59fc8","Type":"ContainerDied","Data":"911eb05950bfbd5fecb1905ff5fc377c7167a48d44a5335b6f07b9623da8bd1d"}
Jul 27 22:23:39.453811 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:39.453786    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:39.546866 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:39.546838    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca\") pod \"8ac9459b-0707-4d97-8405-3043a1a59fc8\" (UID: \"8ac9459b-0707-4d97-8405-3043a1a59fc8\") "
Jul 27 22:23:39.547194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:39.547170    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ac9459b-0707-4d97-8405-3043a1a59fc8" (UID: "8ac9459b-0707-4d97-8405-3043a1a59fc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:23:39.647631 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:39.647547    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac9459b-0707-4d97-8405-3043a1a59fc8-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:23:40.331740 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:40.331702    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg" event={"ID":"8ac9459b-0707-4d97-8405-3043a1a59fc8","Type":"ContainerDied","Data":"c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9"}
Jul 27 22:23:40.331918 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:40.331745    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"
Jul 27 22:23:40.331918 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:40.331759    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c31d77201f0f529125e590a907121f2c612885870cf46237627075362a5c65f9"
Jul 27 22:23:56.225431 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:56.225394    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"]
Jul 27 22:23:56.229506 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:56.229483    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c75b14-zpxcg"]
Jul 27 22:23:56.551937 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:56.551853    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:23:56.916400 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:23:56.916364    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ac9459b-0707-4d97-8405-3043a1a59fc8" path="/var/lib/kubelet/pods/8ac9459b-0707-4d97-8405-3043a1a59fc8/volumes"
Jul 27 22:24:13.755477 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:13.755424    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:24:24.662754 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:24.662717    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:24:37.950176 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.950144    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:24:37.950580 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.950567    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ac9459b-0707-4d97-8405-3043a1a59fc8" containerName="revoke-keys"
Jul 27 22:24:37.950625 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.950582    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ac9459b-0707-4d97-8405-3043a1a59fc8" containerName="revoke-keys"
Jul 27 22:24:37.950667 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.950661    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ac9459b-0707-4d97-8405-3043a1a59fc8" containerName="revoke-keys"
Jul 27 22:24:37.954076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.954058    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.956578 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.956555    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"kube-root-ca.crt\""
Jul 27 22:24:37.957627 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.957607    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"openshift-service-ca.crt\""
Jul 27 22:24:37.957627 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.957619    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"default-dockercfg-jdcn5\""
Jul 27 22:24:37.957792 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.957714    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"auth-test-model-6138cc-kserve-self-signed-certs\""
Jul 27 22:24:37.964230 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.964210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:24:37.992018 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.991987    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.992181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.992033    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.992181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.992059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.992181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.992083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxzkt\" (UniqueName: \"kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.992181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.992164    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:37.992368 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:37.992267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093085 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093241 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093094    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093241 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093120    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093241 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093230    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093408 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxzkt\" (UniqueName: \"kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093408 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093315    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093546 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093420    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093546 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.093673 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.093650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.095547 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.095527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.095806 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.095787    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.102764 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.102743    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxzkt\" (UniqueName: \"kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt\") pod \"auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") " pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.265000 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.264934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:38.401361 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.401331    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:24:38.403528 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:24:38.403493    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf83d5501_f0a8_4cde_9923_3c018e502ed7.slice/crio-d285646a9e47ebb399191b2a52d9bcfe93f8699ccc598e28f22f3ea722f8b14f WatchSource:0}: Error finding container d285646a9e47ebb399191b2a52d9bcfe93f8699ccc598e28f22f3ea722f8b14f: Status 404 returned error can't find the container with id d285646a9e47ebb399191b2a52d9bcfe93f8699ccc598e28f22f3ea722f8b14f
Jul 27 22:24:38.627626 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.627589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" event={"ID":"f83d5501-f0a8-4cde-9923-3c018e502ed7","Type":"ContainerStarted","Data":"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"}
Jul 27 22:24:38.627626 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.627638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" event={"ID":"f83d5501-f0a8-4cde-9923-3c018e502ed7","Type":"ContainerStarted","Data":"d285646a9e47ebb399191b2a52d9bcfe93f8699ccc598e28f22f3ea722f8b14f"}
Jul 27 22:24:38.648088 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:38.648033    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" podStartSLOduration=1.648017457 podStartE2EDuration="1.648017457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:24:38.646239308 +0000 UTC m=+2344.299667962" watchObservedRunningTime="2026-07-27 22:24:38.648017457 +0000 UTC m=+2344.301446138"
Jul 27 22:24:41.450521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:41.450482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:24:48.265677 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:48.265630    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:48.265677 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:48.265684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:48.278198 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:48.278168    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:24:48.691797 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:24:48.691769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:25:13.062052 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.062014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:25:13.065613 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.065585    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.068262 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.068239    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"kube-root-ca.crt\""
Jul 27 22:25:13.069367 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.069345    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"auth-test-model-0c79f1-kserve-self-signed-certs\""
Jul 27 22:25:13.069496 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.069397    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"openshift-service-ca.crt\""
Jul 27 22:25:13.069496 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.069408    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"default-dockercfg-v4dwl\""
Jul 27 22:25:13.081963 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.081930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:25:13.126680 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.126897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.126897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126792    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.126897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llmj\" (UniqueName: \"kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.127059 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.127059 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.126977    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228491 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228587    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228877 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228698    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llmj\" (UniqueName: \"kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228877 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228726    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228987 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228885    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.228987 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.228908    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.229098 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.229033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.230928 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.230910    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.231249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.231230    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.239491 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.239444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llmj\" (UniqueName: \"kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj\") pod \"auth-test-model-0c79f1-kserve-5548674b95-rcpjd\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") " pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.375714 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.375620    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:13.522532 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.522499    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:25:13.525690 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:25:13.525660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod331d9e87_8179_4945_8741_b76ad00c24aa.slice/crio-65a5a7794897832e699d122a90e2af765fab19b86b7c5020ddadd9eda54be4ee WatchSource:0}: Error finding container 65a5a7794897832e699d122a90e2af765fab19b86b7c5020ddadd9eda54be4ee: Status 404 returned error can't find the container with id 65a5a7794897832e699d122a90e2af765fab19b86b7c5020ddadd9eda54be4ee
Jul 27 22:25:13.817252 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.817213    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" event={"ID":"331d9e87-8179-4945-8741-b76ad00c24aa","Type":"ContainerStarted","Data":"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"}
Jul 27 22:25:13.817252 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.817258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" event={"ID":"331d9e87-8179-4945-8741-b76ad00c24aa","Type":"ContainerStarted","Data":"65a5a7794897832e699d122a90e2af765fab19b86b7c5020ddadd9eda54be4ee"}
Jul 27 22:25:13.837319 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:13.837267    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" podStartSLOduration=0.837249608 podStartE2EDuration="837.249608ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:25:13.835655335 +0000 UTC m=+2379.489083978" watchObservedRunningTime="2026-07-27 22:25:13.837249608 +0000 UTC m=+2379.490678249"
Jul 27 22:25:16.472261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:16.472215    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:25:23.376527 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:23.376479    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:23.377133 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:23.376545    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:23.389249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:23.389212    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:23.880089 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:23.880050    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:25:41.363466 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:25:41.363411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:26:02.163769 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:26:02.163687    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:26:16.581186 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:26:16.581142    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:26:28.656809 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:26:28.656768    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:26:43.960516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:26:43.960479    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:27:02.761697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:27:02.761653    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:27:24.456105 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:27:24.456061    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:27:43.166580 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:27:43.166497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:28:09.903708 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:28:09.903669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:28:28.263058 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:28:28.263017    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:28:35.127987 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:28:35.127949    2560 scope.go:117] "RemoveContainer" containerID="9870e63c2212d7e66222ad1715be259d5f30144c894a6d8aa1e899004f153312"
Jul 27 22:28:35.137026 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:28:35.136996    2560 scope.go:117] "RemoveContainer" containerID="7589138cabe280b8e4cae3221df3306d4b9f3cb8c780b99477c03c5d972e02d8"
Jul 27 22:28:52.057298 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:28:52.057261    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:29:10.760613 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:10.760523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:29:32.873833 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:32.873792    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:29:35.163195 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:35.163162    2560 scope.go:117] "RemoveContainer" containerID="b0676443b4299c65a7bc0830cd69164750b358a7469c05d3415da0873dc3cb30"
Jul 27 22:29:37.900332 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:37.900284    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:29:38.836992 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:38.836951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:29:50.058353 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.058316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:29:50.862716 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.862686    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:29:50.866362 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.866340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.869137 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.869109    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6138cc\"/\"sub-test-model-6138cc-kserve-self-signed-certs\""
Jul 27 22:29:50.878229 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.878199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:29:50.936014 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.935975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.936194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.936037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.936194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.936114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.936194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.936182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj6b\" (UniqueName: \"kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.936304 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.936217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:50.936304 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:50.936233    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037353 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037578 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037649 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj6b\" (UniqueName: \"kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037649 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037649 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037814 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037814 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037793    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037931 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037889    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.037979 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.037935    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.040262 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.040225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.040635 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.040614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.048552 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.048522    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj6b\" (UniqueName: \"kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b\") pod \"sub-test-model-6138cc-kserve-666db4d6bc-gxr4x\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") " pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.175670 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.175572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:29:51.326408 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:51.326377    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:29:51.330403 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:29:51.330375    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ce49ac_f71e_4846_9811_2a4e3af907a6.slice/crio-db71b154f4d13f8150b72896f4546c55e66d69e8eadc66ac1c73e202ef4df52b WatchSource:0}: Error finding container db71b154f4d13f8150b72896f4546c55e66d69e8eadc66ac1c73e202ef4df52b: Status 404 returned error can't find the container with id db71b154f4d13f8150b72896f4546c55e66d69e8eadc66ac1c73e202ef4df52b
Jul 27 22:29:52.274770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:52.274714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" event={"ID":"55ce49ac-f71e-4846-9811-2a4e3af907a6","Type":"ContainerStarted","Data":"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"}
Jul 27 22:29:52.274770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:52.274765    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" event={"ID":"55ce49ac-f71e-4846-9811-2a4e3af907a6","Type":"ContainerStarted","Data":"db71b154f4d13f8150b72896f4546c55e66d69e8eadc66ac1c73e202ef4df52b"}
Jul 27 22:29:52.295964 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:52.295902    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" podStartSLOduration=2.295884419 podStartE2EDuration="2.295884419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:29:52.29320637 +0000 UTC m=+2657.946635013" watchObservedRunningTime="2026-07-27 22:29:52.295884419 +0000 UTC m=+2657.949313061"
Jul 27 22:29:52.920521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:52.920481    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" podUID="f83d5501-f0a8-4cde-9923-3c018e502ed7" containerName="main" containerID="cri-o://a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7" gracePeriod=45
Jul 27 22:29:53.177752 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.177674    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:29:53.260817 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.260779    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.260989 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.260887    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.260989 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.260939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxzkt\" (UniqueName: \"kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.260989 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.260976    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.261124 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.261124 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261041    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir\") pod \"f83d5501-f0a8-4cde-9923-3c018e502ed7\" (UID: \"f83d5501-f0a8-4cde-9923-3c018e502ed7\") "
Jul 27 22:29:53.261124 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261105    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:53.261345 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261316    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home" (OuterVolumeSpecName: "home") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:53.261345 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261327    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:53.261499 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.261373    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.263530 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.263501    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt" (OuterVolumeSpecName: "kube-api-access-fxzkt") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "kube-api-access-fxzkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:29:53.263718 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.263699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm" (OuterVolumeSpecName: "dshm") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:53.263804 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.263729    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f83d5501-f0a8-4cde-9923-3c018e502ed7" (UID: "f83d5501-f0a8-4cde-9923-3c018e502ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:29:53.281392 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.281354    2560 generic.go:356] "Generic (PLEG): container finished" podID="f83d5501-f0a8-4cde-9923-3c018e502ed7" containerID="a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7" exitCode=0
Jul 27 22:29:53.281813 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.281402    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"
Jul 27 22:29:53.281813 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.281433    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" event={"ID":"f83d5501-f0a8-4cde-9923-3c018e502ed7","Type":"ContainerDied","Data":"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"}
Jul 27 22:29:53.281813 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.281503    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv" event={"ID":"f83d5501-f0a8-4cde-9923-3c018e502ed7","Type":"ContainerDied","Data":"d285646a9e47ebb399191b2a52d9bcfe93f8699ccc598e28f22f3ea722f8b14f"}
Jul 27 22:29:53.281813 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.281522    2560 scope.go:117] "RemoveContainer" containerID="a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"
Jul 27 22:29:53.291965 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.291939    2560 scope.go:117] "RemoveContainer" containerID="a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"
Jul 27 22:29:53.292303 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:29:53.292276    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7\": container with ID starting with a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7 not found: ID does not exist" containerID="a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"
Jul 27 22:29:53.292388 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.292314    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7"} err="failed to get container status \"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7\": rpc error: code = NotFound desc = could not find container \"a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7\": container with ID starting with a0b1d8334422e987acfb197474a7d5d7fed443077ae0520bdfbcd36e9a27b5a7 not found: ID does not exist"
Jul 27 22:29:53.306648 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.306616    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:29:53.315814 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.315781    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/auth-test-model-6138cc-kserve-7bc6bf58f9-6n9xv"]
Jul 27 22:29:53.362083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.362051    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.362083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.362081    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxzkt\" (UniqueName: \"kubernetes.io/projected/f83d5501-f0a8-4cde-9923-3c018e502ed7-kube-api-access-fxzkt\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.362296 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.362095    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f83d5501-f0a8-4cde-9923-3c018e502ed7-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.362296 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.362109    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.362296 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.362119    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f83d5501-f0a8-4cde-9923-3c018e502ed7-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:53.856925 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.856879    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" containerName="main" containerID="cri-o://48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511" gracePeriod=45
Jul 27 22:29:53.869516 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:53.869439    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 27 22:29:54.102057 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.102018    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:29:54.271799 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.271760    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272010 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.271833    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272010 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.271891    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272010 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.271929    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5llmj\" (UniqueName: \"kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272010 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.271997    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272223 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm\") pod \"331d9e87-8179-4945-8741-b76ad00c24aa\" (UID: \"331d9e87-8179-4945-8741-b76ad00c24aa\") "
Jul 27 22:29:54.272223 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272080    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home" (OuterVolumeSpecName: "home") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:54.272223 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:54.272373 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272302    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:54.272425 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272371    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.272425 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272388    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.272425 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.272398    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.274417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.274393    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm" (OuterVolumeSpecName: "dshm") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:29:54.274417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.274408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:29:54.274595 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.274429    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj" (OuterVolumeSpecName: "kube-api-access-5llmj") pod "331d9e87-8179-4945-8741-b76ad00c24aa" (UID: "331d9e87-8179-4945-8741-b76ad00c24aa"). InnerVolumeSpecName "kube-api-access-5llmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:29:54.289822 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.289790    2560 generic.go:356] "Generic (PLEG): container finished" podID="331d9e87-8179-4945-8741-b76ad00c24aa" containerID="48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511" exitCode=0
Jul 27 22:29:54.290209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.289841    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"
Jul 27 22:29:54.290209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.289843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" event={"ID":"331d9e87-8179-4945-8741-b76ad00c24aa","Type":"ContainerDied","Data":"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"}
Jul 27 22:29:54.290209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.289960    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd" event={"ID":"331d9e87-8179-4945-8741-b76ad00c24aa","Type":"ContainerDied","Data":"65a5a7794897832e699d122a90e2af765fab19b86b7c5020ddadd9eda54be4ee"}
Jul 27 22:29:54.290209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.289982    2560 scope.go:117] "RemoveContainer" containerID="48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"
Jul 27 22:29:54.298994 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.298963    2560 scope.go:117] "RemoveContainer" containerID="48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"
Jul 27 22:29:54.299401 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:29:54.299378    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511\": container with ID starting with 48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511 not found: ID does not exist" containerID="48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"
Jul 27 22:29:54.299504 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.299409    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511"} err="failed to get container status \"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511\": rpc error: code = NotFound desc = could not find container \"48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511\": container with ID starting with 48c8abdf0c7ed1fcdcc0f1833d6a1315f66bdb4ff2dc390aa1dde54ec5b10511 not found: ID does not exist"
Jul 27 22:29:54.315581 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.315544    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:29:54.322571 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.322544    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/auth-test-model-0c79f1-kserve-5548674b95-rcpjd"]
Jul 27 22:29:54.373067 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.373040    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/331d9e87-8179-4945-8741-b76ad00c24aa-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.373067 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.373065    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/331d9e87-8179-4945-8741-b76ad00c24aa-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.373067 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.373074    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5llmj\" (UniqueName: \"kubernetes.io/projected/331d9e87-8179-4945-8741-b76ad00c24aa-kube-api-access-5llmj\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:29:54.918543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.918497    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" path="/var/lib/kubelet/pods/331d9e87-8179-4945-8741-b76ad00c24aa/volumes"
Jul 27 22:29:54.919176 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:29:54.919149    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f83d5501-f0a8-4cde-9923-3c018e502ed7" path="/var/lib/kubelet/pods/f83d5501-f0a8-4cde-9923-3c018e502ed7/volumes"
Jul 27 22:30:00.137465 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137413    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"]
Jul 27 22:30:00.137887 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137853    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f83d5501-f0a8-4cde-9923-3c018e502ed7" containerName="main"
Jul 27 22:30:00.137887 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137869    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f83d5501-f0a8-4cde-9923-3c018e502ed7" containerName="main"
Jul 27 22:30:00.137887 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137880    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" containerName="main"
Jul 27 22:30:00.137887 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137886    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" containerName="main"
Jul 27 22:30:00.138021 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137951    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f83d5501-f0a8-4cde-9923-3c018e502ed7" containerName="main"
Jul 27 22:30:00.138021 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.137962    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="331d9e87-8179-4945-8741-b76ad00c24aa" containerName="main"
Jul 27 22:30:00.142493 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.142441    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"
Jul 27 22:30:00.142970 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.142948    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"]
Jul 27 22:30:00.144973 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.144955    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:30:00.146314 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.146283    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"]
Jul 27 22:30:00.146438 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.146397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"
Jul 27 22:30:00.150045 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.150007    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"]
Jul 27 22:30:00.150194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.150136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"
Jul 27 22:30:00.156443 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.156250    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"
Jul 27 22:30:00.156959 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.156930    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"]
Jul 27 22:30:00.159757 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.159726    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"]
Jul 27 22:30:00.167753 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.167029    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"
Jul 27 22:30:00.169157 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.169131    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:30:00.174237 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.174213    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"
Jul 27 22:30:00.338948 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.338907    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"]
Jul 27 22:30:00.340203 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:30:00.340167    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643e575d_5e4c_4ccc_bb4a_9467a2506ae1.slice/crio-3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3 WatchSource:0}: Error finding container 3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3: Status 404 returned error can't find the container with id 3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3
Jul 27 22:30:00.571158 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.571133    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"]
Jul 27 22:30:00.573439 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:30:00.573412    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d37d472_25a1_4aba_9d05_297891d8713e.slice/crio-42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923 WatchSource:0}: Error finding container 42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923: Status 404 returned error can't find the container with id 42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923
Jul 27 22:30:00.576879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:00.576848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"]
Jul 27 22:30:00.579432 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:30:00.579409    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386dc151_e376_4fd0_9c98_78537aea189b.slice/crio-f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1 WatchSource:0}: Error finding container f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1: Status 404 returned error can't find the container with id f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1
Jul 27 22:30:01.176244 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.176206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:30:01.176656 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.176257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:30:01.189209 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.189182    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:30:01.332298 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.332266    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4" event={"ID":"9d37d472-25a1-4aba-9d05-297891d8713e","Type":"ContainerStarted","Data":"8517ffba04cc40da400a3fe5e511969592516859591cc27ac3379535e19807dd"}
Jul 27 22:30:01.332502 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.332318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4" event={"ID":"9d37d472-25a1-4aba-9d05-297891d8713e","Type":"ContainerStarted","Data":"42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923"}
Jul 27 22:30:01.333819 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.333782    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt" event={"ID":"386dc151-e376-4fd0-9c98-78537aea189b","Type":"ContainerStarted","Data":"f14e70ceee2bc9412d2be1d34e9f283f1ac2fe329bd19041ef8f964a645936f9"}
Jul 27 22:30:01.333948 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.333835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt" event={"ID":"386dc151-e376-4fd0-9c98-78537aea189b","Type":"ContainerStarted","Data":"f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1"}
Jul 27 22:30:01.335231 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.335205    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b" event={"ID":"643e575d-5e4c-4ccc-bb4a-9467a2506ae1","Type":"ContainerStarted","Data":"726b6040cd705ba25284bddb3ca5286070f9879aa2e7fb35e9e0beba5c9b60ed"}
Jul 27 22:30:01.335321 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.335252    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b" event={"ID":"643e575d-5e4c-4ccc-bb4a-9467a2506ae1","Type":"ContainerStarted","Data":"3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3"}
Jul 27 22:30:01.346801 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.346769    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:30:01.349690 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.349645    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4" podStartSLOduration=1.349631746 podStartE2EDuration="1.349631746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:01.348494531 +0000 UTC m=+2667.001923176" watchObservedRunningTime="2026-07-27 22:30:01.349631746 +0000 UTC m=+2667.003060388"
Jul 27 22:30:01.381656 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.381592    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b" podStartSLOduration=1.381570886 podStartE2EDuration="1.381570886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:01.381181399 +0000 UTC m=+2667.034610042" watchObservedRunningTime="2026-07-27 22:30:01.381570886 +0000 UTC m=+2667.034999529"
Jul 27 22:30:01.396188 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:01.396127    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt" podStartSLOduration=1.396090593 podStartE2EDuration="1.396090593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:01.395960199 +0000 UTC m=+2667.049388838" watchObservedRunningTime="2026-07-27 22:30:01.396090593 +0000 UTC m=+2667.049519235"
Jul 27 22:30:02.341296 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.341257    2560 generic.go:356] "Generic (PLEG): container finished" podID="9d37d472-25a1-4aba-9d05-297891d8713e" containerID="8517ffba04cc40da400a3fe5e511969592516859591cc27ac3379535e19807dd" exitCode=0
Jul 27 22:30:02.341759 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.341340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4" event={"ID":"9d37d472-25a1-4aba-9d05-297891d8713e","Type":"ContainerDied","Data":"8517ffba04cc40da400a3fe5e511969592516859591cc27ac3379535e19807dd"}
Jul 27 22:30:02.342821 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.342801    2560 generic.go:356] "Generic (PLEG): container finished" podID="386dc151-e376-4fd0-9c98-78537aea189b" containerID="f14e70ceee2bc9412d2be1d34e9f283f1ac2fe329bd19041ef8f964a645936f9" exitCode=0
Jul 27 22:30:02.342908 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.342828    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt" event={"ID":"386dc151-e376-4fd0-9c98-78537aea189b","Type":"ContainerDied","Data":"f14e70ceee2bc9412d2be1d34e9f283f1ac2fe329bd19041ef8f964a645936f9"}
Jul 27 22:30:02.344249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.344229    2560 generic.go:356] "Generic (PLEG): container finished" podID="643e575d-5e4c-4ccc-bb4a-9467a2506ae1" containerID="726b6040cd705ba25284bddb3ca5286070f9879aa2e7fb35e9e0beba5c9b60ed" exitCode=0
Jul 27 22:30:02.344424 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:02.344250    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b" event={"ID":"643e575d-5e4c-4ccc-bb4a-9467a2506ae1","Type":"ContainerDied","Data":"726b6040cd705ba25284bddb3ca5286070f9879aa2e7fb35e9e0beba5c9b60ed"}
Jul 27 22:30:03.520307 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:03.520279    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"
Jul 27 22:30:03.523529 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:03.523506    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"
Jul 27 22:30:03.550170 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:03.550145    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"
Jul 27 22:30:04.356865 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.356832    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4"
Jul 27 22:30:04.357071 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.356881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753190-hnpd4" event={"ID":"9d37d472-25a1-4aba-9d05-297891d8713e","Type":"ContainerDied","Data":"42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923"}
Jul 27 22:30:04.357071 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.356929    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42b3648ff3d21c29e5e26a1f49f2d6ee268708948e643d1f9680001d5e15f923"
Jul 27 22:30:04.358197 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.358168    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"
Jul 27 22:30:04.358197 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.358175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt" event={"ID":"386dc151-e376-4fd0-9c98-78537aea189b","Type":"ContainerDied","Data":"f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1"}
Jul 27 22:30:04.358397 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.358221    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2a69f4eb416cde2325fc8069603b76cd46c6a96735d0e04de114cdb4b50bce1"
Jul 27 22:30:04.359556 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.359530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b" event={"ID":"643e575d-5e4c-4ccc-bb4a-9467a2506ae1","Type":"ContainerDied","Data":"3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3"}
Jul 27 22:30:04.359556 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.359550    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"
Jul 27 22:30:04.359706 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:04.359563    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e964ffceff68e52188f7d2661253597ea5e2de45dc63669b03a602819cec2f3"
Jul 27 22:30:25.737484 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737783    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="643e575d-5e4c-4ccc-bb4a-9467a2506ae1" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737795    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="643e575d-5e4c-4ccc-bb4a-9467a2506ae1" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737810    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d37d472-25a1-4aba-9d05-297891d8713e" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737815    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d37d472-25a1-4aba-9d05-297891d8713e" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737823    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386dc151-e376-4fd0-9c98-78537aea189b" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737828    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="386dc151-e376-4fd0-9c98-78537aea189b" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737895    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d37d472-25a1-4aba-9d05-297891d8713e" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737904    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="386dc151-e376-4fd0-9c98-78537aea189b" containerName="cleanup"
Jul 27 22:30:25.737914 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.737912    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="643e575d-5e4c-4ccc-bb4a-9467a2506ae1" containerName="cleanup"
Jul 27 22:30:25.741054 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.741032    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.743719 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.743700    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"default-dockercfg-v4dwl\""
Jul 27 22:30:25.744859 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.744833    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"kube-root-ca.crt\""
Jul 27 22:30:25.744964 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.744930    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"sub-test-model-0c79f1-kserve-self-signed-certs\""
Jul 27 22:30:25.745021 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.744974    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"openshift-service-ca.crt\""
Jul 27 22:30:25.759822 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.759791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:30:25.873878 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.873837    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.873878 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.873881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.874090 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.873982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.874090 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.874024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpx6\" (UniqueName: \"kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.874090 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.874081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.874195 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.874103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.974909 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.974862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975119 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.974939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975119 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975025    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpx6\" (UniqueName: \"kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975234    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975390 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975289    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975390 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975552 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.975683 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.975644    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.977956 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.977928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.978238 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.978222    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:25.984983 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:25.984955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpx6\" (UniqueName: \"kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6\") pod \"sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") " pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:26.053078 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:26.052983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:26.206213 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:26.206192    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:30:26.208723 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:30:26.208683    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2027f2a9_581d_412b_bb28_f87c2d8ccb9f.slice/crio-f02cd0a6c99d4cffad8fbf0f518b8cec5b2ad8bac00c769f2738548e94270bfd WatchSource:0}: Error finding container f02cd0a6c99d4cffad8fbf0f518b8cec5b2ad8bac00c769f2738548e94270bfd: Status 404 returned error can't find the container with id f02cd0a6c99d4cffad8fbf0f518b8cec5b2ad8bac00c769f2738548e94270bfd
Jul 27 22:30:26.478429 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:26.478393    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" event={"ID":"2027f2a9-581d-412b-bb28-f87c2d8ccb9f","Type":"ContainerStarted","Data":"2cf84085974df2541476fff64f4f3dd5b4e48f3f23439fee1b49fdbe5f21107d"}
Jul 27 22:30:26.478429 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:26.478436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" event={"ID":"2027f2a9-581d-412b-bb28-f87c2d8ccb9f","Type":"ContainerStarted","Data":"f02cd0a6c99d4cffad8fbf0f518b8cec5b2ad8bac00c769f2738548e94270bfd"}
Jul 27 22:30:26.499147 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:26.499091    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" podStartSLOduration=1.499062826 podStartE2EDuration="1.499062826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:30:26.495913248 +0000 UTC m=+2692.149341890" watchObservedRunningTime="2026-07-27 22:30:26.499062826 +0000 UTC m=+2692.152491469"
Jul 27 22:30:29.558647 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:29.558598    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:30:35.183765 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:35.183726    2560 scope.go:117] "RemoveContainer" containerID="911eb05950bfbd5fecb1905ff5fc377c7167a48d44a5335b6f07b9623da8bd1d"
Jul 27 22:30:36.053882 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:36.053841    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:36.054076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:36.053903    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:36.066575 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:36.066540    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:36.544885 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:36.544843    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:30:58.162205 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:30:58.162160    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:31:16.756701 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:16.756666    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:31:37.450148 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:37.450111    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:31:41.903491 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:41.903433    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:31:42.748799 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:42.748767    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:31:55.052724 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:55.052632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:31:56.923742 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:56.923675    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" podUID="55ce49ac-f71e-4846-9811-2a4e3af907a6" containerName="main" containerID="cri-o://b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315" gracePeriod=45
Jul 27 22:31:57.170260 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.170230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:31:57.264320 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264320 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264320 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264290    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264328    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264384    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzj6b\" (UniqueName: \"kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264421    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache\") pod \"55ce49ac-f71e-4846-9811-2a4e3af907a6\" (UID: \"55ce49ac-f71e-4846-9811-2a4e3af907a6\") "
Jul 27 22:31:57.264661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264629    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home" (OuterVolumeSpecName: "home") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:57.264870 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264668    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:57.264870 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264780    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:57.264972 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264891    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.264972 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264910    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.264972 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.264923    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.266740 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.266711    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm" (OuterVolumeSpecName: "dshm") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:57.266858 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.266765    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b" (OuterVolumeSpecName: "kube-api-access-xzj6b") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "kube-api-access-xzj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:31:57.266858 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.266814    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ce49ac-f71e-4846-9811-2a4e3af907a6" (UID: "55ce49ac-f71e-4846-9811-2a4e3af907a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:31:57.366155 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.366108    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj6b\" (UniqueName: \"kubernetes.io/projected/55ce49ac-f71e-4846-9811-2a4e3af907a6-kube-api-access-xzj6b\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.366155 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.366147    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ce49ac-f71e-4846-9811-2a4e3af907a6-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.366155 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.366160    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ce49ac-f71e-4846-9811-2a4e3af907a6-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:57.768544 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.768482    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" podUID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" containerName="main" containerID="cri-o://2cf84085974df2541476fff64f4f3dd5b4e48f3f23439fee1b49fdbe5f21107d" gracePeriod=45
Jul 27 22:31:57.977040 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.976995    2560 generic.go:356] "Generic (PLEG): container finished" podID="55ce49ac-f71e-4846-9811-2a4e3af907a6" containerID="b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315" exitCode=0
Jul 27 22:31:57.977511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.977093    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"
Jul 27 22:31:57.977511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.977136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" event={"ID":"55ce49ac-f71e-4846-9811-2a4e3af907a6","Type":"ContainerDied","Data":"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"}
Jul 27 22:31:57.977511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.977184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x" event={"ID":"55ce49ac-f71e-4846-9811-2a4e3af907a6","Type":"ContainerDied","Data":"db71b154f4d13f8150b72896f4546c55e66d69e8eadc66ac1c73e202ef4df52b"}
Jul 27 22:31:57.977511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.977233    2560 scope.go:117] "RemoveContainer" containerID="b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"
Jul 27 22:31:57.979569 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.979502    2560 generic.go:356] "Generic (PLEG): container finished" podID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" containerID="2cf84085974df2541476fff64f4f3dd5b4e48f3f23439fee1b49fdbe5f21107d" exitCode=0
Jul 27 22:31:57.979645 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.979562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" event={"ID":"2027f2a9-581d-412b-bb28-f87c2d8ccb9f","Type":"ContainerDied","Data":"2cf84085974df2541476fff64f4f3dd5b4e48f3f23439fee1b49fdbe5f21107d"}
Jul 27 22:31:57.989426 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.989403    2560 scope.go:117] "RemoveContainer" containerID="b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"
Jul 27 22:31:57.989731 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:31:57.989713    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315\": container with ID starting with b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315 not found: ID does not exist" containerID="b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"
Jul 27 22:31:57.989801 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:57.989739    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315"} err="failed to get container status \"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315\": rpc error: code = NotFound desc = could not find container \"b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315\": container with ID starting with b7e5283ae7b704ad1dc3d485c4fc5ff01ecbad62ea3c037b3c7d1b4b51a73315 not found: ID does not exist"
Jul 27 22:31:58.003307 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.003272    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:31:58.007259 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.007234    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6138cc/sub-test-model-6138cc-kserve-666db4d6bc-gxr4x"]
Jul 27 22:31:58.023870 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.023796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:31:58.178406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178443    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178508    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178528    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178606    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178639    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twpx6\" (UniqueName: \"kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6\") pod \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\" (UID: \"2027f2a9-581d-412b-bb28-f87c2d8ccb9f\") "
Jul 27 22:31:58.178902 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178696    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:58.178902 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178761    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:58.178902 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178773    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home" (OuterVolumeSpecName: "home") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:58.179064 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.178984    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.179064 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.179005    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.179064 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.179014    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.180947 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.180915    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:31:58.181069 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.180972    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm" (OuterVolumeSpecName: "dshm") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:31:58.181069 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.180992    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6" (OuterVolumeSpecName: "kube-api-access-twpx6") pod "2027f2a9-581d-412b-bb28-f87c2d8ccb9f" (UID: "2027f2a9-581d-412b-bb28-f87c2d8ccb9f"). InnerVolumeSpecName "kube-api-access-twpx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:31:58.280648 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.280543    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.280648 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.280581    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.280648 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.280596    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpx6\" (UniqueName: \"kubernetes.io/projected/2027f2a9-581d-412b-bb28-f87c2d8ccb9f-kube-api-access-twpx6\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:31:58.915310 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.915272    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ce49ac-f71e-4846-9811-2a4e3af907a6" path="/var/lib/kubelet/pods/55ce49ac-f71e-4846-9811-2a4e3af907a6/volumes"
Jul 27 22:31:58.990492 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.990441    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"
Jul 27 22:31:58.990492 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.990474    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9" event={"ID":"2027f2a9-581d-412b-bb28-f87c2d8ccb9f","Type":"ContainerDied","Data":"f02cd0a6c99d4cffad8fbf0f518b8cec5b2ad8bac00c769f2738548e94270bfd"}
Jul 27 22:31:58.991044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:58.990532    2560 scope.go:117] "RemoveContainer" containerID="2cf84085974df2541476fff64f4f3dd5b4e48f3f23439fee1b49fdbe5f21107d"
Jul 27 22:31:59.009141 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:59.009102    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:31:59.015826 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:31:59.015793    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/sub-test-model-0c79f1-kserve-f6f8ff5-7rvt9"]
Jul 27 22:32:00.916341 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:00.916305    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" path="/var/lib/kubelet/pods/2027f2a9-581d-412b-bb28-f87c2d8ccb9f/volumes"
Jul 27 22:32:04.757008 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:04.756972    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:32:17.854226 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854189    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854587    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ce49ac-f71e-4846-9811-2a4e3af907a6" containerName="main"
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854598    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ce49ac-f71e-4846-9811-2a4e3af907a6" containerName="main"
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854615    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" containerName="main"
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854621    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" containerName="main"
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854681    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ce49ac-f71e-4846-9811-2a4e3af907a6" containerName="main"
Jul 27 22:32:17.854693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.854694    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2027f2a9-581d-412b-bb28-f87c2d8ccb9f" containerName="main"
Jul 27 22:32:17.858079 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.858051    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.860787 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.860758    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"kube-root-ca.crt\""
Jul 27 22:32:17.861760 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.861730    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"rate-test-model-0c79f1-kserve-self-signed-certs\""
Jul 27 22:32:17.861760 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.861749    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"default-dockercfg-v4dwl\""
Jul 27 22:32:17.861928 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.861802    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0c79f1\"/\"openshift-service-ca.crt\""
Jul 27 22:32:17.867354 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.867326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:32:17.961860 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.961822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.962042 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.961908    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.962042 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.961937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtkd\" (UniqueName: \"kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.962042 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.961962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.962152 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.962045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:17.962152 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:17.962084    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062598 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062598 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062690    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtkd\" (UniqueName: \"kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.062879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.062739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.063097 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.063036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.063141 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.063095    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.063177 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.063153    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.065214 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.065192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.065446 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.065426    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.072210 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.072186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtkd\" (UniqueName: \"kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd\") pod \"rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") " pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.169094 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.168992    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:18.310751 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:18.310726    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:32:18.312566 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:32:18.312535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda79206c9_5ca3_4152_8c8f_a07361773e58.slice/crio-01ae9b3ec18c3940b5bd4d8aa0d516e211b8763e85c2b4dd7103659da818508e WatchSource:0}: Error finding container 01ae9b3ec18c3940b5bd4d8aa0d516e211b8763e85c2b4dd7103659da818508e: Status 404 returned error can't find the container with id 01ae9b3ec18c3940b5bd4d8aa0d516e211b8763e85c2b4dd7103659da818508e
Jul 27 22:32:19.103437 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:19.103399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" event={"ID":"a79206c9-5ca3-4152-8c8f-a07361773e58","Type":"ContainerStarted","Data":"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"}
Jul 27 22:32:19.103853 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:19.103477    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" event={"ID":"a79206c9-5ca3-4152-8c8f-a07361773e58","Type":"ContainerStarted","Data":"01ae9b3ec18c3940b5bd4d8aa0d516e211b8763e85c2b4dd7103659da818508e"}
Jul 27 22:32:19.122682 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:19.122626    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" podStartSLOduration=2.122608071 podStartE2EDuration="2.122608071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:32:19.121293882 +0000 UTC m=+2804.774722538" watchObservedRunningTime="2026-07-27 22:32:19.122608071 +0000 UTC m=+2804.776036789"
Jul 27 22:32:21.953222 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:21.953185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:32:28.169511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:28.169465    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:28.170114 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:28.169526    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:28.182775 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:28.182742    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:29.167739 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:29.167706    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:32:51.466514 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:32:51.466470    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:33:09.559405 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:09.559357    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:33:30.259354 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:30.259267    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:33:37.724686 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:37.724642    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:33:38.710198 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.710162    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347787877168"]
Jul 27 22:33:38.713899 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.713878    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:38.716791 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.716748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-548n5\""
Jul 27 22:33:38.718733 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.718703    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787877168"]
Jul 27 22:33:38.846734 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.846695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ptl\" (UniqueName: \"kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl\") pod \"test-curl-7670-140347787877168\" (UID: \"d5b94704-b6d8-4a73-9d48-cc9561c1e978\") " pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:38.948111 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.948071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ptl\" (UniqueName: \"kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl\") pod \"test-curl-7670-140347787877168\" (UID: \"d5b94704-b6d8-4a73-9d48-cc9561c1e978\") " pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:38.957314 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:38.957280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ptl\" (UniqueName: \"kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl\") pod \"test-curl-7670-140347787877168\" (UID: \"d5b94704-b6d8-4a73-9d48-cc9561c1e978\") " pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:39.025068 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:39.024973    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:39.167739 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:39.167712    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787877168"]
Jul 27 22:33:39.170317 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:39.170287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5b94704_b6d8_4a73_9d48_cc9561c1e978.slice/crio-ce35fbcd2f7e9dfa7167e55be1a7c41ac1eb2c27285b217a441fc9a642ae5476 WatchSource:0}: Error finding container ce35fbcd2f7e9dfa7167e55be1a7c41ac1eb2c27285b217a441fc9a642ae5476: Status 404 returned error can't find the container with id ce35fbcd2f7e9dfa7167e55be1a7c41ac1eb2c27285b217a441fc9a642ae5476
Jul 27 22:33:39.171620 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:39.171598    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:33:39.540898 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:39.540855    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787877168" event={"ID":"d5b94704-b6d8-4a73-9d48-cc9561c1e978","Type":"ContainerStarted","Data":"ce35fbcd2f7e9dfa7167e55be1a7c41ac1eb2c27285b217a441fc9a642ae5476"}
Jul 27 22:33:40.546901 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:40.546864    2560 generic.go:356] "Generic (PLEG): container finished" podID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" containerID="d9e334aaea0f84035a44ef52a448fdbc34c5b9cafef02fc4c5568db93658c546" exitCode=0
Jul 27 22:33:40.547321 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:40.546947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787877168" event={"ID":"d5b94704-b6d8-4a73-9d48-cc9561c1e978","Type":"ContainerDied","Data":"d9e334aaea0f84035a44ef52a448fdbc34c5b9cafef02fc4c5568db93658c546"}
Jul 27 22:33:41.688283 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.688249    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:41.760955 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.760929    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347787877168_d5b94704-b6d8-4a73-9d48-cc9561c1e978/test-curl-7670-140347787877168/0.log"
Jul 27 22:33:41.777233 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.777198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9ptl\" (UniqueName: \"kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl\") pod \"d5b94704-b6d8-4a73-9d48-cc9561c1e978\" (UID: \"d5b94704-b6d8-4a73-9d48-cc9561c1e978\") "
Jul 27 22:33:41.779642 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.779609    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl" (OuterVolumeSpecName: "kube-api-access-h9ptl") pod "d5b94704-b6d8-4a73-9d48-cc9561c1e978" (UID: "d5b94704-b6d8-4a73-9d48-cc9561c1e978"). InnerVolumeSpecName "kube-api-access-h9ptl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:41.785929 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.785880    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347787877168"]
Jul 27 22:33:41.789442 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.789412    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347787877168"]
Jul 27 22:33:41.878856 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.878817    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ptl\" (UniqueName: \"kubernetes.io/projected/d5b94704-b6d8-4a73-9d48-cc9561c1e978-kube-api-access-h9ptl\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:41.927058 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.927023    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347787874352"]
Jul 27 22:33:41.927429 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.927413    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" containerName="test-curl-7670-140347787877168"
Jul 27 22:33:41.927498 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.927432    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" containerName="test-curl-7670-140347787877168"
Jul 27 22:33:41.927533 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.927518    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" containerName="test-curl-7670-140347787877168"
Jul 27 22:33:41.930635 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.930616    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:41.950435 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.950405    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787874352"]
Jul 27 22:33:41.979886 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:41.979851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk96v\" (UniqueName: \"kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v\") pod \"test-curl-7670-140347787874352\" (UID: \"52314083-576f-43b9-a403-47e3fb42b44e\") " pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:42.080997 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.080940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk96v\" (UniqueName: \"kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v\") pod \"test-curl-7670-140347787874352\" (UID: \"52314083-576f-43b9-a403-47e3fb42b44e\") " pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:42.090293 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.090256    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk96v\" (UniqueName: \"kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v\") pod \"test-curl-7670-140347787874352\" (UID: \"52314083-576f-43b9-a403-47e3fb42b44e\") " pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:42.240393 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.240308    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:42.383704 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.383669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787874352"]
Jul 27 22:33:42.385932 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:42.385897    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52314083_576f_43b9_a403_47e3fb42b44e.slice/crio-4f8cdc7872ab17f369d8e51d3ac95981f94599c06cb50c28c200e4bfaa09bc61 WatchSource:0}: Error finding container 4f8cdc7872ab17f369d8e51d3ac95981f94599c06cb50c28c200e4bfaa09bc61: Status 404 returned error can't find the container with id 4f8cdc7872ab17f369d8e51d3ac95981f94599c06cb50c28c200e4bfaa09bc61
Jul 27 22:33:42.558857 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.558830    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:42.559057 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.558828    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce35fbcd2f7e9dfa7167e55be1a7c41ac1eb2c27285b217a441fc9a642ae5476"
Jul 27 22:33:42.560042 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.560016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787874352" event={"ID":"52314083-576f-43b9-a403-47e3fb42b44e","Type":"ContainerStarted","Data":"4f8cdc7872ab17f369d8e51d3ac95981f94599c06cb50c28c200e4bfaa09bc61"}
Jul 27 22:33:42.569464 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:33:42.569406    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7670-140347787877168\" is forbidden: User \"system:node:ip-10-0-143-110.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-110.ec2.internal' and this object" podUID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" pod="opendatahub/test-curl-7670-140347787877168"
Jul 27 22:33:42.917631 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:42.917535    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b94704-b6d8-4a73-9d48-cc9561c1e978" path="/var/lib/kubelet/pods/d5b94704-b6d8-4a73-9d48-cc9561c1e978/volumes"
Jul 27 22:33:43.566901 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:43.566862    2560 generic.go:356] "Generic (PLEG): container finished" podID="52314083-576f-43b9-a403-47e3fb42b44e" containerID="fc1acaadf100742994578010abf60b04bcebb54e665d834f725b3bf5bf186546" exitCode=0
Jul 27 22:33:43.567107 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:43.566962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787874352" event={"ID":"52314083-576f-43b9-a403-47e3fb42b44e","Type":"ContainerDied","Data":"fc1acaadf100742994578010abf60b04bcebb54e665d834f725b3bf5bf186546"}
Jul 27 22:33:44.461829 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.461791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:33:44.710504 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.710477    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:44.731194 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.731108    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347787874352_52314083-576f-43b9-a403-47e3fb42b44e/test-curl-7670-140347787874352/0.log"
Jul 27 22:33:44.754786 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.754746    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347787874352"]
Jul 27 22:33:44.758977 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.758944    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347787874352"]
Jul 27 22:33:44.807441 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.807404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk96v\" (UniqueName: \"kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v\") pod \"52314083-576f-43b9-a403-47e3fb42b44e\" (UID: \"52314083-576f-43b9-a403-47e3fb42b44e\") "
Jul 27 22:33:44.809772 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.809739    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v" (OuterVolumeSpecName: "kube-api-access-wk96v") pod "52314083-576f-43b9-a403-47e3fb42b44e" (UID: "52314083-576f-43b9-a403-47e3fb42b44e"). InnerVolumeSpecName "kube-api-access-wk96v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:44.908306 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.908263    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk96v\" (UniqueName: \"kubernetes.io/projected/52314083-576f-43b9-a403-47e3fb42b44e-kube-api-access-wk96v\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:44.916468 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.916424    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52314083-576f-43b9-a403-47e3fb42b44e" path="/var/lib/kubelet/pods/52314083-576f-43b9-a403-47e3fb42b44e/volumes"
Jul 27 22:33:44.936851 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.936813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:44.937230 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.937216    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52314083-576f-43b9-a403-47e3fb42b44e" containerName="test-curl-7670-140347787874352"
Jul 27 22:33:44.937288 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.937232    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="52314083-576f-43b9-a403-47e3fb42b44e" containerName="test-curl-7670-140347787874352"
Jul 27 22:33:44.937324 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.937304    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="52314083-576f-43b9-a403-47e3fb42b44e" containerName="test-curl-7670-140347787874352"
Jul 27 22:33:44.940341 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.940322    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:44.947995 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:44.947968    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:45.009117 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.009008    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxrt\" (UniqueName: \"kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt\") pod \"test-curl-7670-140347785279360\" (UID: \"426a2423-764e-4fa0-b432-302ee4f8a161\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:45.110348 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.110305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxrt\" (UniqueName: \"kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt\") pod \"test-curl-7670-140347785279360\" (UID: \"426a2423-764e-4fa0-b432-302ee4f8a161\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:45.119478 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.119411    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxrt\" (UniqueName: \"kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt\") pod \"test-curl-7670-140347785279360\" (UID: \"426a2423-764e-4fa0-b432-302ee4f8a161\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:45.251138 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.251102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:45.395412 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.395364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:45.398189 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:45.398164    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod426a2423_764e_4fa0_b432_302ee4f8a161.slice/crio-555c57eb1ab5d1011d7d1d6fa0f7e3a23c710ba9700f9fe8e640d3e7e8c5d6f2 WatchSource:0}: Error finding container 555c57eb1ab5d1011d7d1d6fa0f7e3a23c710ba9700f9fe8e640d3e7e8c5d6f2: Status 404 returned error can't find the container with id 555c57eb1ab5d1011d7d1d6fa0f7e3a23c710ba9700f9fe8e640d3e7e8c5d6f2
Jul 27 22:33:45.580092 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.580059    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787874352"
Jul 27 22:33:45.580595 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.580065    2560 scope.go:117] "RemoveContainer" containerID="fc1acaadf100742994578010abf60b04bcebb54e665d834f725b3bf5bf186546"
Jul 27 22:33:45.581493 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:45.581376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347785279360" event={"ID":"426a2423-764e-4fa0-b432-302ee4f8a161","Type":"ContainerStarted","Data":"555c57eb1ab5d1011d7d1d6fa0f7e3a23c710ba9700f9fe8e640d3e7e8c5d6f2"}
Jul 27 22:33:46.590354 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:46.590319    2560 generic.go:356] "Generic (PLEG): container finished" podID="426a2423-764e-4fa0-b432-302ee4f8a161" containerID="b515ee34450395cf3bb496be17016e70526df71ef822d6ec95ffeff093bcfbb6" exitCode=0
Jul 27 22:33:46.590849 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:46.590412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347785279360" event={"ID":"426a2423-764e-4fa0-b432-302ee4f8a161","Type":"ContainerDied","Data":"b515ee34450395cf3bb496be17016e70526df71ef822d6ec95ffeff093bcfbb6"}
Jul 27 22:33:47.738804 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.738773    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:47.763206 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.763179    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347785279360_426a2423-764e-4fa0-b432-302ee4f8a161/test-curl-7670-140347785279360/0.log"
Jul 27 22:33:47.790750 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.790714    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:47.793177 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.793149    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:47.837610 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.837575    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sxrt\" (UniqueName: \"kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt\") pod \"426a2423-764e-4fa0-b432-302ee4f8a161\" (UID: \"426a2423-764e-4fa0-b432-302ee4f8a161\") "
Jul 27 22:33:47.839866 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.839838    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt" (OuterVolumeSpecName: "kube-api-access-7sxrt") pod "426a2423-764e-4fa0-b432-302ee4f8a161" (UID: "426a2423-764e-4fa0-b432-302ee4f8a161"). InnerVolumeSpecName "kube-api-access-7sxrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:47.916384 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.916298    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:47.916774 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.916757    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426a2423-764e-4fa0-b432-302ee4f8a161" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:47.916815 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.916776    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="426a2423-764e-4fa0-b432-302ee4f8a161" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:47.916868 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.916859    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="426a2423-764e-4fa0-b432-302ee4f8a161" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:47.919974 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.919954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:47.927115 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.927075    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:47.938989 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:47.938956    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sxrt\" (UniqueName: \"kubernetes.io/projected/426a2423-764e-4fa0-b432-302ee4f8a161-kube-api-access-7sxrt\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:48.040288 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.040235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftgz\" (UniqueName: \"kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz\") pod \"test-curl-7670-140347785279360\" (UID: \"462ea345-b6f0-4668-a624-aa6d555a549a\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:48.141832 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.141783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftgz\" (UniqueName: \"kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz\") pod \"test-curl-7670-140347785279360\" (UID: \"462ea345-b6f0-4668-a624-aa6d555a549a\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:48.154319 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.154288    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftgz\" (UniqueName: \"kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz\") pod \"test-curl-7670-140347785279360\" (UID: \"462ea345-b6f0-4668-a624-aa6d555a549a\") " pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:48.231622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.231518    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:48.381477 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.381422    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:48.384352 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:48.384322    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod462ea345_b6f0_4668_a624_aa6d555a549a.slice/crio-24292ab7d9fb1d37644e06b959b0331a0ba05e0dde7bfa5497484728859e7f97 WatchSource:0}: Error finding container 24292ab7d9fb1d37644e06b959b0331a0ba05e0dde7bfa5497484728859e7f97: Status 404 returned error can't find the container with id 24292ab7d9fb1d37644e06b959b0331a0ba05e0dde7bfa5497484728859e7f97
Jul 27 22:33:48.612115 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.612077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347785279360" event={"ID":"462ea345-b6f0-4668-a624-aa6d555a549a","Type":"ContainerStarted","Data":"24292ab7d9fb1d37644e06b959b0331a0ba05e0dde7bfa5497484728859e7f97"}
Jul 27 22:33:48.613673 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.613627    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="555c57eb1ab5d1011d7d1d6fa0f7e3a23c710ba9700f9fe8e640d3e7e8c5d6f2"
Jul 27 22:33:48.613833 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.613688    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:48.626184 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.626126    2560 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7670-140347785279360" oldPodUID="426a2423-764e-4fa0-b432-302ee4f8a161" podUID="462ea345-b6f0-4668-a624-aa6d555a549a"
Jul 27 22:33:48.916013 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:48.915929    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426a2423-764e-4fa0-b432-302ee4f8a161" path="/var/lib/kubelet/pods/426a2423-764e-4fa0-b432-302ee4f8a161/volumes"
Jul 27 22:33:49.620357 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:49.620325    2560 generic.go:356] "Generic (PLEG): container finished" podID="462ea345-b6f0-4668-a624-aa6d555a549a" containerID="820658f2817bd4dde69185924db2131b0dcd28230778f7256b6bbf42aecca8ad" exitCode=0
Jul 27 22:33:49.620553 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:49.620411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347785279360" event={"ID":"462ea345-b6f0-4668-a624-aa6d555a549a","Type":"ContainerDied","Data":"820658f2817bd4dde69185924db2131b0dcd28230778f7256b6bbf42aecca8ad"}
Jul 27 22:33:50.766300 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.766268    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:50.788504 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.788475    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347785279360_462ea345-b6f0-4668-a624-aa6d555a549a/test-curl-7670-140347785279360/0.log"
Jul 27 22:33:50.812443 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.812403    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:50.820064 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.820033    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347785279360"]
Jul 27 22:33:50.869184 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.869148    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wftgz\" (UniqueName: \"kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz\") pod \"462ea345-b6f0-4668-a624-aa6d555a549a\" (UID: \"462ea345-b6f0-4668-a624-aa6d555a549a\") "
Jul 27 22:33:50.871409 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.871379    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz" (OuterVolumeSpecName: "kube-api-access-wftgz") pod "462ea345-b6f0-4668-a624-aa6d555a549a" (UID: "462ea345-b6f0-4668-a624-aa6d555a549a"). InnerVolumeSpecName "kube-api-access-wftgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:50.915980 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.915939    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="462ea345-b6f0-4668-a624-aa6d555a549a" path="/var/lib/kubelet/pods/462ea345-b6f0-4668-a624-aa6d555a549a/volumes"
Jul 27 22:33:50.946491 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.946431    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:50.946843 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.946830    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="462ea345-b6f0-4668-a624-aa6d555a549a" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:50.946897 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.946845    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="462ea345-b6f0-4668-a624-aa6d555a549a" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:50.946939 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.946914    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="462ea345-b6f0-4668-a624-aa6d555a549a" containerName="test-curl-7670-140347785279360"
Jul 27 22:33:50.950056 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.950036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:50.956375 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.956347    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:50.970633 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:50.970588    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftgz\" (UniqueName: \"kubernetes.io/projected/462ea345-b6f0-4668-a624-aa6d555a549a-kube-api-access-wftgz\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:51.071958 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.071917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjhb\" (UniqueName: \"kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb\") pod \"test-curl-7670-140347787892320\" (UID: \"c2ed40ff-09b0-4369-8aa8-d74f8363bf26\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:51.173241 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.173127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjhb\" (UniqueName: \"kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb\") pod \"test-curl-7670-140347787892320\" (UID: \"c2ed40ff-09b0-4369-8aa8-d74f8363bf26\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:51.183271 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.183235    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjhb\" (UniqueName: \"kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb\") pod \"test-curl-7670-140347787892320\" (UID: \"c2ed40ff-09b0-4369-8aa8-d74f8363bf26\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:51.262651 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.262607    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:51.407817 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.407781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:51.409759 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:51.409719    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2ed40ff_09b0_4369_8aa8_d74f8363bf26.slice/crio-307e95c5923e3f05ac643884746809142214cde4a53f64177ec1277bfd32b664 WatchSource:0}: Error finding container 307e95c5923e3f05ac643884746809142214cde4a53f64177ec1277bfd32b664: Status 404 returned error can't find the container with id 307e95c5923e3f05ac643884746809142214cde4a53f64177ec1277bfd32b664
Jul 27 22:33:51.633730 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.633694    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347785279360"
Jul 27 22:33:51.633933 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.633743    2560 scope.go:117] "RemoveContainer" containerID="820658f2817bd4dde69185924db2131b0dcd28230778f7256b6bbf42aecca8ad"
Jul 27 22:33:51.635024 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:51.634972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787892320" event={"ID":"c2ed40ff-09b0-4369-8aa8-d74f8363bf26","Type":"ContainerStarted","Data":"307e95c5923e3f05ac643884746809142214cde4a53f64177ec1277bfd32b664"}
Jul 27 22:33:52.641419 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:52.641374    2560 generic.go:356] "Generic (PLEG): container finished" podID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" containerID="f1e8e7db61b23c347b4d9abb9ecb94b6e844e121d4edddd5def40cc86d59d476" exitCode=0
Jul 27 22:33:52.641868 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:52.641472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787892320" event={"ID":"c2ed40ff-09b0-4369-8aa8-d74f8363bf26","Type":"ContainerDied","Data":"f1e8e7db61b23c347b4d9abb9ecb94b6e844e121d4edddd5def40cc86d59d476"}
Jul 27 22:33:52.744971 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:52.744921    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" podUID="a79206c9-5ca3-4152-8c8f-a07361773e58" containerName="main" containerID="cri-o://e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc" gracePeriod=45
Jul 27 22:33:53.020091 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.020062    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:33:53.093153 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093111    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093164    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093193    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093229    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vtkd\" (UniqueName: \"kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093331 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093271    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm\") pod \"a79206c9-5ca3-4152-8c8f-a07361773e58\" (UID: \"a79206c9-5ca3-4152-8c8f-a07361773e58\") "
Jul 27 22:33:53.093683 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home" (OuterVolumeSpecName: "home") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:33:53.093683 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093654    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.093683 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093648    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache" (OuterVolumeSpecName: "model-cache") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:33:53.093846 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.093828    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:33:53.095788 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.095751    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd" (OuterVolumeSpecName: "kube-api-access-6vtkd") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "kube-api-access-6vtkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:53.095891 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.095820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:33:53.096130 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.096101    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm" (OuterVolumeSpecName: "dshm") pod "a79206c9-5ca3-4152-8c8f-a07361773e58" (UID: "a79206c9-5ca3-4152-8c8f-a07361773e58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:33:53.195011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.194918    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.195011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.194949    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79206c9-5ca3-4152-8c8f-a07361773e58-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.195011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.194958    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtkd\" (UniqueName: \"kubernetes.io/projected/a79206c9-5ca3-4152-8c8f-a07361773e58-kube-api-access-6vtkd\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.195011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.194967    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.195011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.194977    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79206c9-5ca3-4152-8c8f-a07361773e58-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:53.650916 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.650885    2560 generic.go:356] "Generic (PLEG): container finished" podID="a79206c9-5ca3-4152-8c8f-a07361773e58" containerID="e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc" exitCode=0
Jul 27 22:33:53.651333 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.650929    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"
Jul 27 22:33:53.651333 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.650975    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" event={"ID":"a79206c9-5ca3-4152-8c8f-a07361773e58","Type":"ContainerDied","Data":"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"}
Jul 27 22:33:53.651333 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.651022    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5" event={"ID":"a79206c9-5ca3-4152-8c8f-a07361773e58","Type":"ContainerDied","Data":"01ae9b3ec18c3940b5bd4d8aa0d516e211b8763e85c2b4dd7103659da818508e"}
Jul 27 22:33:53.651333 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.651043    2560 scope.go:117] "RemoveContainer" containerID="e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"
Jul 27 22:33:53.661122 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.661098    2560 scope.go:117] "RemoveContainer" containerID="e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"
Jul 27 22:33:53.661509 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:33:53.661484    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc\": container with ID starting with e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc not found: ID does not exist" containerID="e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"
Jul 27 22:33:53.661654 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.661517    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc"} err="failed to get container status \"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc\": rpc error: code = NotFound desc = could not find container \"e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc\": container with ID starting with e7c3344bcee94d37c5e2c6656bcc52fc4ca0703f91733523e174765b90fe10cc not found: ID does not exist"
Jul 27 22:33:53.673995 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.673962    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:33:53.680342 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.680309    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0c79f1/rate-test-model-0c79f1-kserve-6b875dbf87-2l2x5"]
Jul 27 22:33:53.790133 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.790101    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:53.814429 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.814402    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347787892320_c2ed40ff-09b0-4369-8aa8-d74f8363bf26/test-curl-7670-140347787892320/0.log"
Jul 27 22:33:53.844614 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.844579    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:53.848941 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.848904    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:53.900463 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.900428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cjhb\" (UniqueName: \"kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb\") pod \"c2ed40ff-09b0-4369-8aa8-d74f8363bf26\" (UID: \"c2ed40ff-09b0-4369-8aa8-d74f8363bf26\") "
Jul 27 22:33:53.902779 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.902717    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb" (OuterVolumeSpecName: "kube-api-access-4cjhb") pod "c2ed40ff-09b0-4369-8aa8-d74f8363bf26" (UID: "c2ed40ff-09b0-4369-8aa8-d74f8363bf26"). InnerVolumeSpecName "kube-api-access-4cjhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:53.978230 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978194    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:53.978633 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978619    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a79206c9-5ca3-4152-8c8f-a07361773e58" containerName="main"
Jul 27 22:33:53.978684 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978635    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a79206c9-5ca3-4152-8c8f-a07361773e58" containerName="main"
Jul 27 22:33:53.978684 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978650    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" containerName="test-curl-7670-140347787892320"
Jul 27 22:33:53.978684 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978660    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" containerName="test-curl-7670-140347787892320"
Jul 27 22:33:53.978781 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978716    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a79206c9-5ca3-4152-8c8f-a07361773e58" containerName="main"
Jul 27 22:33:53.978781 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.978729    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" containerName="test-curl-7670-140347787892320"
Jul 27 22:33:53.981795 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.981775    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:53.986858 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:53.986829    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:54.001729 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.001689    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjhb\" (UniqueName: \"kubernetes.io/projected/c2ed40ff-09b0-4369-8aa8-d74f8363bf26-kube-api-access-4cjhb\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:54.102927 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.102889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42zn4\" (UniqueName: \"kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4\") pod \"test-curl-7670-140347787892320\" (UID: \"73da673c-4225-4674-a74d-5911296491be\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:54.204604 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.204498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42zn4\" (UniqueName: \"kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4\") pod \"test-curl-7670-140347787892320\" (UID: \"73da673c-4225-4674-a74d-5911296491be\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:54.213852 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.213822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42zn4\" (UniqueName: \"kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4\") pod \"test-curl-7670-140347787892320\" (UID: \"73da673c-4225-4674-a74d-5911296491be\") " pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:54.292881 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.292835    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:54.436893 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.436865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:54.438948 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:33:54.438919    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73da673c_4225_4674_a74d_5911296491be.slice/crio-c8c1e59dcaa3ca4116af3acd59f87023e7c2bb406b4a8d90412a60d747747836 WatchSource:0}: Error finding container c8c1e59dcaa3ca4116af3acd59f87023e7c2bb406b4a8d90412a60d747747836: Status 404 returned error can't find the container with id c8c1e59dcaa3ca4116af3acd59f87023e7c2bb406b4a8d90412a60d747747836
Jul 27 22:33:54.659546 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.659510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787892320" event={"ID":"73da673c-4225-4674-a74d-5911296491be","Type":"ContainerStarted","Data":"c8c1e59dcaa3ca4116af3acd59f87023e7c2bb406b4a8d90412a60d747747836"}
Jul 27 22:33:54.660747 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.660727    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307e95c5923e3f05ac643884746809142214cde4a53f64177ec1277bfd32b664"
Jul 27 22:33:54.660747 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.660743    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:54.673430 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.673399    2560 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7670-140347787892320" oldPodUID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" podUID="73da673c-4225-4674-a74d-5911296491be"
Jul 27 22:33:54.916822 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.916785    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a79206c9-5ca3-4152-8c8f-a07361773e58" path="/var/lib/kubelet/pods/a79206c9-5ca3-4152-8c8f-a07361773e58/volumes"
Jul 27 22:33:54.917261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:54.917237    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2ed40ff-09b0-4369-8aa8-d74f8363bf26" path="/var/lib/kubelet/pods/c2ed40ff-09b0-4369-8aa8-d74f8363bf26/volumes"
Jul 27 22:33:55.667676 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:55.667639    2560 generic.go:356] "Generic (PLEG): container finished" podID="73da673c-4225-4674-a74d-5911296491be" containerID="0c7a63a561ec04cca9eb9333b651114e61232f3d4da4327d2afa609418278900" exitCode=0
Jul 27 22:33:55.668094 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:55.667728    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7670-140347787892320" event={"ID":"73da673c-4225-4674-a74d-5911296491be","Type":"ContainerDied","Data":"0c7a63a561ec04cca9eb9333b651114e61232f3d4da4327d2afa609418278900"}
Jul 27 22:33:56.821056 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.821024    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:56.842520 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.842489    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7670-140347787892320_73da673c-4225-4674-a74d-5911296491be/test-curl-7670-140347787892320/0.log"
Jul 27 22:33:56.866705 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.866665    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:56.873000 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.872953    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7670-140347787892320"]
Jul 27 22:33:56.932649 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.932554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42zn4\" (UniqueName: \"kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4\") pod \"73da673c-4225-4674-a74d-5911296491be\" (UID: \"73da673c-4225-4674-a74d-5911296491be\") "
Jul 27 22:33:56.934794 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:56.934767    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4" (OuterVolumeSpecName: "kube-api-access-42zn4") pod "73da673c-4225-4674-a74d-5911296491be" (UID: "73da673c-4225-4674-a74d-5911296491be"). InnerVolumeSpecName "kube-api-access-42zn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:33:57.033829 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:57.033786    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42zn4\" (UniqueName: \"kubernetes.io/projected/73da673c-4225-4674-a74d-5911296491be-kube-api-access-42zn4\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:33:57.680882 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:57.680849    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7670-140347787892320"
Jul 27 22:33:57.681076 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:57.680886    2560 scope.go:117] "RemoveContainer" containerID="0c7a63a561ec04cca9eb9333b651114e61232f3d4da4327d2afa609418278900"
Jul 27 22:33:58.915974 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:33:58.915937    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73da673c-4225-4674-a74d-5911296491be" path="/var/lib/kubelet/pods/73da673c-4225-4674-a74d-5911296491be/volumes"
Jul 27 22:34:01.604096 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.604053    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:34:01.604521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.604440    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73da673c-4225-4674-a74d-5911296491be" containerName="test-curl-7670-140347787892320"
Jul 27 22:34:01.604521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.604465    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="73da673c-4225-4674-a74d-5911296491be" containerName="test-curl-7670-140347787892320"
Jul 27 22:34:01.604646 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.604536    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="73da673c-4225-4674-a74d-5911296491be" containerName="test-curl-7670-140347787892320"
Jul 27 22:34:01.609077 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.609055    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.611899 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.611875    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-626801-dockercfg-gfrxh\""
Jul 27 22:34:01.612185 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.612168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-626801\""
Jul 27 22:34:01.620200 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.620159    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:34:01.685424 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.685387    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:34:01.688993 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.688969    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.701328 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.701299    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:34:01.782414 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.782367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.782659 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.782547    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.782659 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.782594    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v86h9\" (UniqueName: \"kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.782750 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.782674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fms7\" (UniqueName: \"kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.883384 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.883273    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.883384 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.883327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v86h9\" (UniqueName: \"kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.883384 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.883348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fms7\" (UniqueName: \"kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.883384 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.883390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.884026 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.884005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.884160 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.884140    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.892253 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.892221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fms7\" (UniqueName: \"kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7\") pod \"payload-processing-e2e-ipp-a-626801-69fdb47896-8454z\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:01.892420 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.892221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v86h9\" (UniqueName: \"kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9\") pod \"payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:01.922337 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:01.922290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:02.001035 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:02.000998    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:02.084484 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:02.084439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:34:02.087042 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:02.087015    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0866802_d54b_4de4_aa33_b8b42e8af928.slice/crio-f599e5faf1c50846940fda59daa78b5127a7a3eca3f8a55cb582b2d8bed26231 WatchSource:0}: Error finding container f599e5faf1c50846940fda59daa78b5127a7a3eca3f8a55cb582b2d8bed26231: Status 404 returned error can't find the container with id f599e5faf1c50846940fda59daa78b5127a7a3eca3f8a55cb582b2d8bed26231
Jul 27 22:34:02.155517 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:02.155476    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:34:02.164629 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:02.158235    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7577b6d_17c1_472c_8323_dc945446d277.slice/crio-ade7db12a110f83baf657fdf25484b07f54d104ce1978ea79098a8ba7048db8a WatchSource:0}: Error finding container ade7db12a110f83baf657fdf25484b07f54d104ce1978ea79098a8ba7048db8a: Status 404 returned error can't find the container with id ade7db12a110f83baf657fdf25484b07f54d104ce1978ea79098a8ba7048db8a
Jul 27 22:34:02.716197 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:02.716065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" event={"ID":"c0866802-d54b-4de4-aa33-b8b42e8af928","Type":"ContainerStarted","Data":"f599e5faf1c50846940fda59daa78b5127a7a3eca3f8a55cb582b2d8bed26231"}
Jul 27 22:34:02.718789 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:02.718740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" event={"ID":"a7577b6d-17c1-472c-8323-dc945446d277","Type":"ContainerStarted","Data":"ade7db12a110f83baf657fdf25484b07f54d104ce1978ea79098a8ba7048db8a"}
Jul 27 22:34:04.732225 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.732185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" event={"ID":"c0866802-d54b-4de4-aa33-b8b42e8af928","Type":"ContainerStarted","Data":"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"}
Jul 27 22:34:04.732731 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.732302    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:04.733951 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.733923    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" event={"ID":"a7577b6d-17c1-472c-8323-dc945446d277","Type":"ContainerStarted","Data":"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"}
Jul 27 22:34:04.734083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.734024    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:04.752091 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.752043    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" podStartSLOduration=2.003476498 podStartE2EDuration="3.752027745s" totalImagesPullingTime="1.748551247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:04.750644658 +0000 UTC m=+2910.404073302" watchObservedRunningTime="2026-07-27 22:34:04.752027745 +0000 UTC m=+2910.405456384"
Jul 27 22:34:04.769277 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:04.769225    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" podStartSLOduration=2.091360287 podStartE2EDuration="3.769206275s" totalImagesPullingTime="1.677845988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:04.767712583 +0000 UTC m=+2910.421141237" watchObservedRunningTime="2026-07-27 22:34:04.769206275 +0000 UTC m=+2910.422634917"
Jul 27 22:34:09.857325 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:09.857284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:34:15.742275 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:15.742237    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:34:15.742744 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:15.742324    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:34:21.543684 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.543642    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:34:21.549173 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.549145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.551814 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.551784    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-adca75-openshift-default-dockercfg-btfqk\""
Jul 27 22:34:21.551988 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.551783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 22:34:21.560923 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.560891    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:34:21.669743 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.669934 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669766    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.669934 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxzg\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.669934 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.669934 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.669934 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.669913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.670102 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.670001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.670102 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.670027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.670102 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.670069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771380 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771330    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771380 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771382    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771533    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771565    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.771707 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771634    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxzg\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.772089 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.771971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.772089 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.772031    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.772261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.772099    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.772261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.772131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.772261 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.772239    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.774014 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.773992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.774346 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.774322    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.780135 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.780102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.780250 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.780154    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxzg\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg\") pod \"e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") " pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:21.861964 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:21.861915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:22.030161 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:22.030135    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:34:22.032093 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:22.032064    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7d5d50_8686_47e4_9861_5c42356ab49d.slice/crio-601d8ab66d9b4a43749f6c1cdc97a89d2cb927542b5477b91e85203e7133212c WatchSource:0}: Error finding container 601d8ab66d9b4a43749f6c1cdc97a89d2cb927542b5477b91e85203e7133212c: Status 404 returned error can't find the container with id 601d8ab66d9b4a43749f6c1cdc97a89d2cb927542b5477b91e85203e7133212c
Jul 27 22:34:22.827738 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:22.827699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerStarted","Data":"601d8ab66d9b4a43749f6c1cdc97a89d2cb927542b5477b91e85203e7133212c"}
Jul 27 22:34:24.830969 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:24.830932    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:24.831302 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:24.831020    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:24.831302 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:24.831052    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:25.845110 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:25.845072    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerStarted","Data":"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"}
Jul 27 22:34:25.862359 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:25.862326    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:25.868332 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:25.868305    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:25.868685 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:25.868642    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" podStartSLOduration=2.072349377 podStartE2EDuration="4.868628054s" totalImagesPullingTime="2.796278677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:25.866611842 +0000 UTC m=+2931.520040509" watchObservedRunningTime="2026-07-27 22:34:25.868628054 +0000 UTC m=+2931.522056695"
Jul 27 22:34:26.853869 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:26.853830    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:26.858213 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:26.858184    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:28.809364 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.809319    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:34:28.813174 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.813152    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:28.817439 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.817409    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-adca75-dockercfg-smrvv\""
Jul 27 22:34:28.817745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.817723    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-adca75\""
Jul 27 22:34:28.825438 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.825409    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:34:28.833775 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.833743    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:34:28.837446 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.837419    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:28.849870 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.849839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:34:28.945418 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.945369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:28.945418 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.945411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk49z\" (UniqueName: \"kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:28.945712 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.945596    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:28.945712 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:28.945638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfqs\" (UniqueName: \"kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.046788 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.046730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.046788 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.046788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfqs\" (UniqueName: \"kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.047007 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.046841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.047007 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.046860    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk49z\" (UniqueName: \"kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.047704 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.047676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.047831 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.047814    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.059173 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.059139    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk49z\" (UniqueName: \"kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z\") pod \"payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.059521 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.059418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfqs\" (UniqueName: \"kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs\") pod \"payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.125321 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.125279    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.149976 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.149937    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.288721 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.288690    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:34:29.291106 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:29.291076    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c2a9a6_e08f_470b_8bb6_4c3b8d7a37ff.slice/crio-df58936b7a85aaa6d041d4e9c46ad3cb025d21c4197bf76f19c5a93d4c5cd53b WatchSource:0}: Error finding container df58936b7a85aaa6d041d4e9c46ad3cb025d21c4197bf76f19c5a93d4c5cd53b: Status 404 returned error can't find the container with id df58936b7a85aaa6d041d4e9c46ad3cb025d21c4197bf76f19c5a93d4c5cd53b
Jul 27 22:34:29.322673 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.322643    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:34:29.326740 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:29.326710    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e50966e_844e_42f6_ab48_8efbc9085014.slice/crio-901c93d8ca97092932142b152d26e408742b55341bb62c5fddc45565a81b9c1e WatchSource:0}: Error finding container 901c93d8ca97092932142b152d26e408742b55341bb62c5fddc45565a81b9c1e: Status 404 returned error can't find the container with id 901c93d8ca97092932142b152d26e408742b55341bb62c5fddc45565a81b9c1e
Jul 27 22:34:29.870301 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.870260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" event={"ID":"3e50966e-844e-42f6-ab48-8efbc9085014","Type":"ContainerStarted","Data":"bf190dffcf0ede339ae45100739d56740ca1328626fee0e71dbef148174c2047"}
Jul 27 22:34:29.870803 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.870312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" event={"ID":"3e50966e-844e-42f6-ab48-8efbc9085014","Type":"ContainerStarted","Data":"901c93d8ca97092932142b152d26e408742b55341bb62c5fddc45565a81b9c1e"}
Jul 27 22:34:29.870803 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.870340    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:29.872048 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.872026    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" event={"ID":"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff","Type":"ContainerStarted","Data":"23e72066c8dceaff6a3bc5f1be6523527fbfff30e787c5645a80453f0f3cb5dd"}
Jul 27 22:34:29.872176 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.872065    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" event={"ID":"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff","Type":"ContainerStarted","Data":"df58936b7a85aaa6d041d4e9c46ad3cb025d21c4197bf76f19c5a93d4c5cd53b"}
Jul 27 22:34:29.872176 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.872095    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:29.892364 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.892300    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" podStartSLOduration=1.892278631 podStartE2EDuration="1.892278631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:29.888648259 +0000 UTC m=+2935.542077130" watchObservedRunningTime="2026-07-27 22:34:29.892278631 +0000 UTC m=+2935.545707276"
Jul 27 22:34:29.908299 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:29.908237    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" podStartSLOduration=1.908214281 podStartE2EDuration="1.908214281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:29.908030244 +0000 UTC m=+2935.561458938" watchObservedRunningTime="2026-07-27 22:34:29.908214281 +0000 UTC m=+2935.561642927"
Jul 27 22:34:32.765288 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:32.765248    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:34:40.879482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.879389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:34:40.879875 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.879499    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:34:40.955554 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.955437    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerID="a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d" exitCode=0
Jul 27 22:34:40.955554 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.955491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerDied","Data":"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"}
Jul 27 22:34:40.955989 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.955974    2560 scope.go:117] "RemoveContainer" containerID="a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"
Jul 27 22:34:40.957130 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.957099    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:40.957213 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.957168    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:40.957213 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:40.957200    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:34:41.854137 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:41.854092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:41.963358 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:41.963318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerStarted","Data":"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"}
Jul 27 22:34:42.862467 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:42.862424    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:42.863666 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:42.863636    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 27 22:34:42.863727 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:42.863701    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 27 22:34:43.862843 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:43.862798    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 27 22:34:43.863269 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:43.862870    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 27 22:34:44.862440 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:44.862403    2560 patch_prober.go:28] interesting pod/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Jul 27 22:34:44.862661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:44.862482    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Jul 27 22:34:45.866893 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:45.866859    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:45.867362 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:45.867266    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:45.868185 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:45.868164    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:34:49.826250 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:49.826210    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg_3e50966e-844e-42f6-ab48-8efbc9085014/payload-processing/0.log"
Jul 27 22:34:53.353478 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.353407    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:34:53.356159 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.356133    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.358981 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.358955    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-626801\"/\"kube-root-ca.crt\""
Jul 27 22:34:53.360050 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.360018    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-626801\"/\"ipp-route-626801-kserve-self-signed-certs\""
Jul 27 22:34:53.360188 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.360053    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-626801\"/\"openshift-service-ca.crt\""
Jul 27 22:34:53.360188 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.360119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-626801\"/\"default-dockercfg-glhxs\""
Jul 27 22:34:53.367693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.367659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:34:53.410487 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.410710 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfdl\" (UniqueName: \"kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.410771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410699    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.410771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.410856 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.410895 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.410866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512277 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512511 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512419    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfdl\" (UniqueName: \"kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512807 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512752    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512807 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512767    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.512901 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.512819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.514932 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.514897    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.515176 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.515158    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.521560 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.521529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfdl\" (UniqueName: \"kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl\") pod \"ipp-route-626801-kserve-5df667dc7f-qtdzk\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") " pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.668817 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.668705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:34:53.816827 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:53.816787    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:34:53.820556 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:34:53.820518    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7050a989_4cc4_4c11_a3b6_3a5f75fc31fd.slice/crio-604edc72e5655906e34b6b33f45aed389e97c63c0b435f9cadb1869c0b23fd01 WatchSource:0}: Error finding container 604edc72e5655906e34b6b33f45aed389e97c63c0b435f9cadb1869c0b23fd01: Status 404 returned error can't find the container with id 604edc72e5655906e34b6b33f45aed389e97c63c0b435f9cadb1869c0b23fd01
Jul 27 22:34:54.030612 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:54.030516    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" event={"ID":"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd","Type":"ContainerStarted","Data":"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"}
Jul 27 22:34:54.030612 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:54.030569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" event={"ID":"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd","Type":"ContainerStarted","Data":"604edc72e5655906e34b6b33f45aed389e97c63c0b435f9cadb1869c0b23fd01"}
Jul 27 22:34:54.049128 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:54.049071    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" podStartSLOduration=1.049053355 podStartE2EDuration="1.049053355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:34:54.048153243 +0000 UTC m=+2959.701581886" watchObservedRunningTime="2026-07-27 22:34:54.049053355 +0000 UTC m=+2959.702481997"
Jul 27 22:34:57.458311 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:34:57.458222    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:35:03.669188 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:03.669126    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:35:03.669700 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:03.669212    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:35:03.681883 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:03.681851    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:35:04.097790 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:04.097742    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:35:35.059084 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:35.059048    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:35:44.994978 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:44.994947    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-626801-69fdb47896-8454z_a7577b6d-17c1-472c-8323-dc945446d277/payload-processing/0.log"
Jul 27 22:35:45.259654 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:45.259572    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg_3e50966e-844e-42f6-ab48-8efbc9085014/payload-processing/0.log"
Jul 27 22:35:45.880937 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:45.880900    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:35:45.883697 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:45.883672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:45.886303 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:45.886281    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1368f7-openshift-default-dockercfg-j5vxf\""
Jul 27 22:35:45.897272 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:45.897237    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:35:46.042342 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042342 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042587    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.042847 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.042703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4ws\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.143641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143543    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.143641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143580    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.143641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.143641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143682    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4ws\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.143806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144295 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.144029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144295 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.144104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144295 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.144194    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144490 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.144311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.144490 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.144433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.146322 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.146299    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.146820 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.146798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.152544 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.152515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.152661 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.152623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4ws\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws\") pod \"e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") " pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.195504 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.195434    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:46.350902 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.350874    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:35:46.352951 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:35:46.352922    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee143474_491f_4182_8686_bafa7e3ca3c8.slice/crio-e0a6e5cd79fce6974bc28e1d791ec7ed1dd33002c4be23bd2a06083f57a7574a WatchSource:0}: Error finding container e0a6e5cd79fce6974bc28e1d791ec7ed1dd33002c4be23bd2a06083f57a7574a: Status 404 returned error can't find the container with id e0a6e5cd79fce6974bc28e1d791ec7ed1dd33002c4be23bd2a06083f57a7574a
Jul 27 22:35:46.355240 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.355204    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:35:46.355303 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.355282    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:35:46.355345 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:46.355312    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:35:47.335889 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:47.335854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerStarted","Data":"c5db3e1b171010b4e52ea5ed0b773b6537ae2597fd294bd418ca5009cdd1011c"}
Jul 27 22:35:47.336294 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:47.335909    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerStarted","Data":"e0a6e5cd79fce6974bc28e1d791ec7ed1dd33002c4be23bd2a06083f57a7574a"}
Jul 27 22:35:47.362257 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:47.362203    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podStartSLOduration=2.362184114 podStartE2EDuration="2.362184114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:35:47.360144241 +0000 UTC m=+3013.013572908" watchObservedRunningTime="2026-07-27 22:35:47.362184114 +0000 UTC m=+3013.015612756"
Jul 27 22:35:48.196639 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:48.196598    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:48.201926 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:48.201894    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:48.341815 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:48.341779    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:48.342818 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:48.342794    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:35:54.255543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:35:54.255498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:36:03.343868 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.343822    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 27 22:36:03.344300 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.343900    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 27 22:36:03.432181 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.432147    2560 generic.go:356] "Generic (PLEG): container finished" podID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerID="c5db3e1b171010b4e52ea5ed0b773b6537ae2597fd294bd418ca5009cdd1011c" exitCode=0
Jul 27 22:36:03.432366 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.432214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerDied","Data":"c5db3e1b171010b4e52ea5ed0b773b6537ae2597fd294bd418ca5009cdd1011c"}
Jul 27 22:36:03.432701 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.432686    2560 scope.go:117] "RemoveContainer" containerID="c5db3e1b171010b4e52ea5ed0b773b6537ae2597fd294bd418ca5009cdd1011c"
Jul 27 22:36:03.433871 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.433841    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:36:03.433937 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.433919    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:36:03.433993 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:03.433960    2560 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"}
Jul 27 22:36:04.440236 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:04.440192    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerStarted","Data":"a932ccf653bbcb262a4b5d61f2a745fdd71a3e7ef01ef57da07304afdc774268"}
Jul 27 22:36:05.196152 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:05.196106    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:05.196480 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:05.196426    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 27 22:36:05.196597 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:05.196539    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 27 22:36:06.196954 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:06.196424    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 27 22:36:06.196954 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:06.196532    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 27 22:36:07.196578 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:07.196532    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 27 22:36:07.196799 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:07.196605    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 27 22:36:08.196338 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:08.196286    2560 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Jul 27 22:36:08.196883 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:08.196368    2560 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Jul 27 22:36:09.201103 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:09.201067    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:09.201590 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:09.201346    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:09.202121 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:09.202101    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:23.269165 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:23.269120    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:36:26.405977 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:26.405895    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:36:26.406417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:26.406181    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy" containerID="cri-o://a932ccf653bbcb262a4b5d61f2a745fdd71a3e7ef01ef57da07304afdc774268" gracePeriod=30
Jul 27 22:36:31.610266 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.610226    2560 generic.go:356] "Generic (PLEG): container finished" podID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerID="a932ccf653bbcb262a4b5d61f2a745fdd71a3e7ef01ef57da07304afdc774268" exitCode=0
Jul 27 22:36:31.610736 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.610296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerDied","Data":"a932ccf653bbcb262a4b5d61f2a745fdd71a3e7ef01ef57da07304afdc774268"}
Jul 27 22:36:31.610736 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.610380    2560 scope.go:117] "RemoveContainer" containerID="c5db3e1b171010b4e52ea5ed0b773b6537ae2597fd294bd418ca5009cdd1011c"
Jul 27 22:36:31.667596 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.667568    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:31.704592 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704493    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dx4ws\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704592 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704542    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704592 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704569    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704592 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704593    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704623    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704710    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704778    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704841    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert\") pod \"ee143474-491f-4182-8686-bafa7e3ca3c8\" (UID: \"ee143474-491f-4182-8686-bafa7e3ca3c8\") "
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:31.704943 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.704938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:31.705324 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.705158    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.705324 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.705178    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-workload-socket\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.705324 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.705266    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:31.706038 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.706002    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:31.706133 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.706082    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:31.707190 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.707164    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:31.707417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.707400    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:36:31.707568 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.707552    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws" (OuterVolumeSpecName: "kube-api-access-dx4ws") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "kube-api-access-dx4ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:31.708191 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.708163    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee143474-491f-4182-8686-bafa7e3ca3c8" (UID: "ee143474-491f-4182-8686-bafa7e3ca3c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 22:36:31.806193 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806157    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-credential-socket\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806193 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806189    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-data\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806193 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806199    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee143474-491f-4182-8686-bafa7e3ca3c8-istiod-ca-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806207    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4ws\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-kube-api-access-dx4ws\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806216    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-token\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806224    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-podinfo\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:31.806482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:31.806231    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee143474-491f-4182-8686-bafa7e3ca3c8-istio-envoy\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:32.617966 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.617926    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt" event={"ID":"ee143474-491f-4182-8686-bafa7e3ca3c8","Type":"ContainerDied","Data":"e0a6e5cd79fce6974bc28e1d791ec7ed1dd33002c4be23bd2a06083f57a7574a"}
Jul 27 22:36:32.618435 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.617991    2560 scope.go:117] "RemoveContainer" containerID="a932ccf653bbcb262a4b5d61f2a745fdd71a3e7ef01ef57da07304afdc774268"
Jul 27 22:36:32.618435 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.617935    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"
Jul 27 22:36:32.641569 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.641533    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:36:32.645946 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.645915    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1368f7-openshift-default-5b6794dbf4-lcqdt"]
Jul 27 22:36:32.923785 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:32.923687    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" path="/var/lib/kubelet/pods/ee143474-491f-4182-8686-bafa7e3ca3c8/volumes"
Jul 27 22:36:45.827083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:45.827039    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:36:45.827702 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:45.827376    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" podUID="a7577b6d-17c1-472c-8323-dc945446d277" containerName="payload-processing" containerID="cri-o://8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726" gracePeriod=30
Jul 27 22:36:45.835286 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:45.835248    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:36:45.835613 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:45.835554    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" podUID="c0866802-d54b-4de4-aa33-b8b42e8af928" containerName="payload-pre-processing" containerID="cri-o://490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74" gracePeriod=30
Jul 27 22:36:46.130651 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.130621    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:36:46.133888 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.133861    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:36:46.259120 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259086    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v86h9\" (UniqueName: \"kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9\") pod \"c0866802-d54b-4de4-aa33-b8b42e8af928\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") "
Jul 27 22:36:46.259311 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume\") pod \"c0866802-d54b-4de4-aa33-b8b42e8af928\" (UID: \"c0866802-d54b-4de4-aa33-b8b42e8af928\") "
Jul 27 22:36:46.259311 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259195    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume\") pod \"a7577b6d-17c1-472c-8323-dc945446d277\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") "
Jul 27 22:36:46.259311 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259220    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fms7\" (UniqueName: \"kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7\") pod \"a7577b6d-17c1-472c-8323-dc945446d277\" (UID: \"a7577b6d-17c1-472c-8323-dc945446d277\") "
Jul 27 22:36:46.259626 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a7577b6d-17c1-472c-8323-dc945446d277" (UID: "a7577b6d-17c1-472c-8323-dc945446d277"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:46.259709 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.259633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0866802-d54b-4de4-aa33-b8b42e8af928" (UID: "c0866802-d54b-4de4-aa33-b8b42e8af928"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:36:46.261637 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.261610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7" (OuterVolumeSpecName: "kube-api-access-5fms7") pod "a7577b6d-17c1-472c-8323-dc945446d277" (UID: "a7577b6d-17c1-472c-8323-dc945446d277"). InnerVolumeSpecName "kube-api-access-5fms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:46.261637 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.261621    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9" (OuterVolumeSpecName: "kube-api-access-v86h9") pod "c0866802-d54b-4de4-aa33-b8b42e8af928" (UID: "c0866802-d54b-4de4-aa33-b8b42e8af928"). InnerVolumeSpecName "kube-api-access-v86h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:36:46.360745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.360633    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v86h9\" (UniqueName: \"kubernetes.io/projected/c0866802-d54b-4de4-aa33-b8b42e8af928-kube-api-access-v86h9\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.360745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.360672    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0866802-d54b-4de4-aa33-b8b42e8af928-plugins-config-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.360745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.360687    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7577b6d-17c1-472c-8323-dc945446d277-plugins-config-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.360745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.360700    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fms7\" (UniqueName: \"kubernetes.io/projected/a7577b6d-17c1-472c-8323-dc945446d277-kube-api-access-5fms7\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:36:46.698536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.698417    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0866802-d54b-4de4-aa33-b8b42e8af928" containerID="490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74" exitCode=0
Jul 27 22:36:46.698536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.698494    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"
Jul 27 22:36:46.698536 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.698495    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" event={"ID":"c0866802-d54b-4de4-aa33-b8b42e8af928","Type":"ContainerDied","Data":"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"}
Jul 27 22:36:46.698818 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.698556    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd" event={"ID":"c0866802-d54b-4de4-aa33-b8b42e8af928","Type":"ContainerDied","Data":"f599e5faf1c50846940fda59daa78b5127a7a3eca3f8a55cb582b2d8bed26231"}
Jul 27 22:36:46.698818 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.698585    2560 scope.go:117] "RemoveContainer" containerID="490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"
Jul 27 22:36:46.700367 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.700343    2560 generic.go:356] "Generic (PLEG): container finished" podID="a7577b6d-17c1-472c-8323-dc945446d277" containerID="8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726" exitCode=0
Jul 27 22:36:46.700507 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.700387    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"
Jul 27 22:36:46.700507 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.700420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" event={"ID":"a7577b6d-17c1-472c-8323-dc945446d277","Type":"ContainerDied","Data":"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"}
Jul 27 22:36:46.700507 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.700491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z" event={"ID":"a7577b6d-17c1-472c-8323-dc945446d277","Type":"ContainerDied","Data":"ade7db12a110f83baf657fdf25484b07f54d104ce1978ea79098a8ba7048db8a"}
Jul 27 22:36:46.709929 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.709905    2560 scope.go:117] "RemoveContainer" containerID="490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"
Jul 27 22:36:46.710301 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:36:46.710280    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74\": container with ID starting with 490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74 not found: ID does not exist" containerID="490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"
Jul 27 22:36:46.710398 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.710313    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74"} err="failed to get container status \"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74\": rpc error: code = NotFound desc = could not find container \"490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74\": container with ID starting with 490fd05bf4d72d397532905399dd89001c9701966a30a4347d3171c310ae2e74 not found: ID does not exist"
Jul 27 22:36:46.710398 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.710337    2560 scope.go:117] "RemoveContainer" containerID="8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"
Jul 27 22:36:46.719556 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.719531    2560 scope.go:117] "RemoveContainer" containerID="8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"
Jul 27 22:36:46.719870 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:36:46.719850    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726\": container with ID starting with 8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726 not found: ID does not exist" containerID="8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"
Jul 27 22:36:46.719942 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.719878    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726"} err="failed to get container status \"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726\": rpc error: code = NotFound desc = could not find container \"8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726\": container with ID starting with 8ab09c48757659bd65d2bcc8318e301a6326323b22fccee3670a1265e0817726 not found: ID does not exist"
Jul 27 22:36:46.729083 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.729046    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:36:46.733533 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.733503    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-626801-7c5bdd6fd4-9crfd"]
Jul 27 22:36:46.747701 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.747667    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:36:46.749828 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.749803    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-626801-69fdb47896-8454z"]
Jul 27 22:36:46.918110 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.918076    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7577b6d-17c1-472c-8323-dc945446d277" path="/var/lib/kubelet/pods/a7577b6d-17c1-472c-8323-dc945446d277/volumes"
Jul 27 22:36:46.918799 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.918518    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0866802-d54b-4de4-aa33-b8b42e8af928" path="/var/lib/kubelet/pods/c0866802-d54b-4de4-aa33-b8b42e8af928/volumes"
Jul 27 22:36:46.956945 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:46.956851    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:36:55.800118 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:55.800077    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:36:56.455225 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:36:56.455184    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:37:10.821984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:10.821926    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" podUID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" containerName="main" containerID="cri-o://9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516" gracePeriod=45
Jul 27 22:37:11.079499 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.079412    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:37:11.114575 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114539    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114647    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114704    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114727    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxfdl\" (UniqueName: \"kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114770 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114746    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114783    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs\") pod \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\" (UID: \"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd\") "
Jul 27 22:37:11.114984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114859    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home" (OuterVolumeSpecName: "home") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:11.114984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114955    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:11.114984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.114972    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:11.115203 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.115156    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-model-cache\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.115203 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.115173    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tmp-dir\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.115203 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.115187    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-home\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.117791 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.117756    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm" (OuterVolumeSpecName: "dshm") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:37:11.117907 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.117825    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl" (OuterVolumeSpecName: "kube-api-access-sxfdl") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "kube-api-access-sxfdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:11.117907 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.117878    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" (UID: "7050a989-4cc4-4c11-a3b6-3a5f75fc31fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 22:37:11.216482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.216415    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfdl\" (UniqueName: \"kubernetes.io/projected/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-kube-api-access-sxfdl\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.216482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.216465    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-dshm\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.216482 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.216480    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd-tls-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:11.845419 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.845379    2560 generic.go:356] "Generic (PLEG): container finished" podID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" containerID="9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516" exitCode=0
Jul 27 22:37:11.845863 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.845427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"
Jul 27 22:37:11.845863 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.845504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" event={"ID":"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd","Type":"ContainerDied","Data":"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"}
Jul 27 22:37:11.845863 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.845552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk" event={"ID":"7050a989-4cc4-4c11-a3b6-3a5f75fc31fd","Type":"ContainerDied","Data":"604edc72e5655906e34b6b33f45aed389e97c63c0b435f9cadb1869c0b23fd01"}
Jul 27 22:37:11.845863 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.845577    2560 scope.go:117] "RemoveContainer" containerID="9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"
Jul 27 22:37:11.856217 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.856186    2560 scope.go:117] "RemoveContainer" containerID="9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"
Jul 27 22:37:11.856522 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:37:11.856503    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516\": container with ID starting with 9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516 not found: ID does not exist" containerID="9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"
Jul 27 22:37:11.856586 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.856532    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516"} err="failed to get container status \"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516\": rpc error: code = NotFound desc = could not find container \"9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516\": container with ID starting with 9c23e5277f435480e058445d385839eb57821fef5f8cba3891c2db0146c45516 not found: ID does not exist"
Jul 27 22:37:11.870170 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.870134    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:37:11.876927 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:11.876889    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-626801/ipp-route-626801-kserve-5df667dc7f-qtdzk"]
Jul 27 22:37:12.916739 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:12.916703    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" path="/var/lib/kubelet/pods/7050a989-4cc4-4c11-a3b6-3a5f75fc31fd/volumes"
Jul 27 22:37:39.289912 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.289875    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"]
Jul 27 22:37:39.290321 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290310    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7577b6d-17c1-472c-8323-dc945446d277" containerName="payload-processing"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290324    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7577b6d-17c1-472c-8323-dc945446d277" containerName="payload-processing"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290338    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0866802-d54b-4de4-aa33-b8b42e8af928" containerName="payload-pre-processing"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290345    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0866802-d54b-4de4-aa33-b8b42e8af928" containerName="payload-pre-processing"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290353    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" containerName="main"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290359    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" containerName="main"
Jul 27 22:37:39.290369 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290367    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290372    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290382    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290388    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290482    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290491    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7050a989-4cc4-4c11-a3b6-3a5f75fc31fd" containerName="main"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290499    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7577b6d-17c1-472c-8323-dc945446d277" containerName="payload-processing"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290506    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee143474-491f-4182-8686-bafa7e3ca3c8" containerName="istio-proxy"
Jul 27 22:37:39.290641 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.290512    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0866802-d54b-4de4-aa33-b8b42e8af928" containerName="payload-pre-processing"
Jul 27 22:37:39.294791 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.294771    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:39.297470 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.297428    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:37:39.297594 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.297534    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:37:39.300999 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.300972    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"]
Jul 27 22:37:39.398771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.398726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8\" (UID: \"6ba55933-b78f-4a86-ba53-41bb77b5ede8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:39.499827 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.499792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8\" (UID: \"6ba55933-b78f-4a86-ba53-41bb77b5ede8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:39.500568 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.500543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8\" (UID: \"6ba55933-b78f-4a86-ba53-41bb77b5ede8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:39.607476 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.607422    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:39.756928 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:39.756901    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"]
Jul 27 22:37:39.759426 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:37:39.759394    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba55933_b78f_4a86_ba53_41bb77b5ede8.slice/crio-c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda WatchSource:0}: Error finding container c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda: Status 404 returned error can't find the container with id c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda
Jul 27 22:37:40.002478 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:40.002380    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8" event={"ID":"6ba55933-b78f-4a86-ba53-41bb77b5ede8","Type":"ContainerStarted","Data":"c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda"}
Jul 27 22:37:41.009498 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:41.009368    2560 generic.go:356] "Generic (PLEG): container finished" podID="6ba55933-b78f-4a86-ba53-41bb77b5ede8" containerID="3e527b705c3a17f9fc3f9d0aaf68269632f4f4ca5171a920a42f970d6980feca" exitCode=0
Jul 27 22:37:41.009950 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:41.009494    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8" event={"ID":"6ba55933-b78f-4a86-ba53-41bb77b5ede8","Type":"ContainerDied","Data":"3e527b705c3a17f9fc3f9d0aaf68269632f4f4ca5171a920a42f970d6980feca"}
Jul 27 22:37:42.162767 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:42.162736    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:42.327570 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:42.327528    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca\") pod \"6ba55933-b78f-4a86-ba53-41bb77b5ede8\" (UID: \"6ba55933-b78f-4a86-ba53-41bb77b5ede8\") "
Jul 27 22:37:42.327895 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:42.327872    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ba55933-b78f-4a86-ba53-41bb77b5ede8" (UID: "6ba55933-b78f-4a86-ba53-41bb77b5ede8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:42.428838 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:42.428797    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ba55933-b78f-4a86-ba53-41bb77b5ede8-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:43.022051 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:43.022016    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"
Jul 27 22:37:43.022233 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:43.022052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8" event={"ID":"6ba55933-b78f-4a86-ba53-41bb77b5ede8","Type":"ContainerDied","Data":"c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda"}
Jul 27 22:37:43.022233 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:43.022098    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c2dd5c545bb218d3fbc3022d0f21e9f5ead29c93a88891a1df3d35ab10bfda"
Jul 27 22:37:49.835263 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:49.835223    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:37:49.835820 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:49.835516    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" podUID="3e50966e-844e-42f6-ab48-8efbc9085014" containerName="payload-processing" containerID="cri-o://bf190dffcf0ede339ae45100739d56740ca1328626fee0e71dbef148174c2047" gracePeriod=30
Jul 27 22:37:49.844850 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:49.844819    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:37:49.845110 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:49.845078    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" podUID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" containerName="payload-pre-processing" containerID="cri-o://23e72066c8dceaff6a3bc5f1be6523527fbfff30e787c5645a80453f0f3cb5dd" gracePeriod=30
Jul 27 22:37:50.063854 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.063765    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e50966e-844e-42f6-ab48-8efbc9085014" containerID="bf190dffcf0ede339ae45100739d56740ca1328626fee0e71dbef148174c2047" exitCode=0
Jul 27 22:37:50.065262 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.063921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" event={"ID":"3e50966e-844e-42f6-ab48-8efbc9085014","Type":"ContainerDied","Data":"bf190dffcf0ede339ae45100739d56740ca1328626fee0e71dbef148174c2047"}
Jul 27 22:37:50.065975 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.065947    2560 generic.go:356] "Generic (PLEG): container finished" podID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" containerID="23e72066c8dceaff6a3bc5f1be6523527fbfff30e787c5645a80453f0f3cb5dd" exitCode=0
Jul 27 22:37:50.066097 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.066052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" event={"ID":"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff","Type":"ContainerDied","Data":"23e72066c8dceaff6a3bc5f1be6523527fbfff30e787c5645a80453f0f3cb5dd"}
Jul 27 22:37:50.120806 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.120765    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:37:50.144034 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.143957    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:37:50.196553 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.196513    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk49z\" (UniqueName: \"kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z\") pod \"3e50966e-844e-42f6-ab48-8efbc9085014\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") "
Jul 27 22:37:50.196730 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.196630    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcfqs\" (UniqueName: \"kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs\") pod \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") "
Jul 27 22:37:50.196730 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.196685    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume\") pod \"3e50966e-844e-42f6-ab48-8efbc9085014\" (UID: \"3e50966e-844e-42f6-ab48-8efbc9085014\") "
Jul 27 22:37:50.196730 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.196714    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume\") pod \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\" (UID: \"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff\") "
Jul 27 22:37:50.197120 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.197090    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e50966e-844e-42f6-ab48-8efbc9085014" (UID: "3e50966e-844e-42f6-ab48-8efbc9085014"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:50.197406 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.197095    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" (UID: "63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:37:50.198848 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.198814    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z" (OuterVolumeSpecName: "kube-api-access-hk49z") pod "3e50966e-844e-42f6-ab48-8efbc9085014" (UID: "3e50966e-844e-42f6-ab48-8efbc9085014"). InnerVolumeSpecName "kube-api-access-hk49z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:50.198948 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.198907    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs" (OuterVolumeSpecName: "kube-api-access-rcfqs") pod "63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" (UID: "63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff"). InnerVolumeSpecName "kube-api-access-rcfqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:37:50.298276 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.298215    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e50966e-844e-42f6-ab48-8efbc9085014-plugins-config-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.298276 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.298266    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-plugins-config-volume\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.298276 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.298276    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk49z\" (UniqueName: \"kubernetes.io/projected/3e50966e-844e-42f6-ab48-8efbc9085014-kube-api-access-hk49z\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:50.298276 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:50.298284    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfqs\" (UniqueName: \"kubernetes.io/projected/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff-kube-api-access-rcfqs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:37:51.072693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.072661    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"
Jul 27 22:37:51.072693 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.072685    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg" event={"ID":"3e50966e-844e-42f6-ab48-8efbc9085014","Type":"ContainerDied","Data":"901c93d8ca97092932142b152d26e408742b55341bb62c5fddc45565a81b9c1e"}
Jul 27 22:37:51.073233 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.072755    2560 scope.go:117] "RemoveContainer" containerID="bf190dffcf0ede339ae45100739d56740ca1328626fee0e71dbef148174c2047"
Jul 27 22:37:51.074238 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.074212    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md" event={"ID":"63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff","Type":"ContainerDied","Data":"df58936b7a85aaa6d041d4e9c46ad3cb025d21c4197bf76f19c5a93d4c5cd53b"}
Jul 27 22:37:51.074535 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.074245    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"
Jul 27 22:37:51.082417 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.082394    2560 scope.go:117] "RemoveContainer" containerID="23e72066c8dceaff6a3bc5f1be6523527fbfff30e787c5645a80453f0f3cb5dd"
Jul 27 22:37:51.090782 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.090744    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:37:51.094617 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.094588    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-adca75-65f69cc876-978md"]
Jul 27 22:37:51.106233 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.106137    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:37:51.110086 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:51.110050    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-adca75-6c9766db9d-2trbg"]
Jul 27 22:37:52.915651 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:52.915617    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e50966e-844e-42f6-ab48-8efbc9085014" path="/var/lib/kubelet/pods/3e50966e-844e-42f6-ab48-8efbc9085014/volumes"
Jul 27 22:37:52.916037 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:52.915991    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" path="/var/lib/kubelet/pods/63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff/volumes"
Jul 27 22:37:53.265199 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:53.265091    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:37:54.374277 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:54.374237    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"]
Jul 27 22:37:54.378319 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:54.378289    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-adca75-w9jj8"]
Jul 27 22:37:54.915431 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:37:54.915381    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba55933-b78f-4a86-ba53-41bb77b5ede8" path="/var/lib/kubelet/pods/6ba55933-b78f-4a86-ba53-41bb77b5ede8/volumes"
Jul 27 22:38:01.120502 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.120444    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:38:01.120971 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.120685    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy" containerID="cri-o://bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f" gracePeriod=30
Jul 27 22:38:01.544740 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.544625    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"]
Jul 27 22:38:01.545159 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545138    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e50966e-844e-42f6-ab48-8efbc9085014" containerName="payload-processing"
Jul 27 22:38:01.545249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545162    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e50966e-844e-42f6-ab48-8efbc9085014" containerName="payload-processing"
Jul 27 22:38:01.545249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545186    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba55933-b78f-4a86-ba53-41bb77b5ede8" containerName="revoke-keys"
Jul 27 22:38:01.545249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545195    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba55933-b78f-4a86-ba53-41bb77b5ede8" containerName="revoke-keys"
Jul 27 22:38:01.545249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545208    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" containerName="payload-pre-processing"
Jul 27 22:38:01.545249 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545216    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" containerName="payload-pre-processing"
Jul 27 22:38:01.545543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545332    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c2a9a6-e08f-470b-8bb6-4c3b8d7a37ff" containerName="payload-pre-processing"
Jul 27 22:38:01.545543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545349    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e50966e-844e-42f6-ab48-8efbc9085014" containerName="payload-processing"
Jul 27 22:38:01.545543 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.545359    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba55933-b78f-4a86-ba53-41bb77b5ede8" containerName="revoke-keys"
Jul 27 22:38:01.550022 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.549989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:01.552822 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.552793    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:38:01.552971 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.552872    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:38:01.556186 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.556154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"]
Jul 27 22:38:01.608323 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.608274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g\" (UID: \"77c194f6-471b-43e5-b503-6c3941f442c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:01.709663 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.709614    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g\" (UID: \"77c194f6-471b-43e5-b503-6c3941f442c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:01.710269 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.710246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g\" (UID: \"77c194f6-471b-43e5-b503-6c3941f442c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:01.862841 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:01.862797    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:02.009678 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:02.009646    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"]
Jul 27 22:38:02.012939 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:38:02.012905    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c194f6_471b_43e5_b503_6c3941f442c8.slice/crio-2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69 WatchSource:0}: Error finding container 2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69: Status 404 returned error can't find the container with id 2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69
Jul 27 22:38:02.148964 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:02.148864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g" event={"ID":"77c194f6-471b-43e5-b503-6c3941f442c8","Type":"ContainerStarted","Data":"2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69"}
Jul 27 22:38:03.155299 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:03.155261    2560 generic.go:356] "Generic (PLEG): container finished" podID="77c194f6-471b-43e5-b503-6c3941f442c8" containerID="465aac688215ab87897b4179de77e3eeb161d08751228e042b0cba27dffa5775" exitCode=0
Jul 27 22:38:03.155704 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:03.155336    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g" event={"ID":"77c194f6-471b-43e5-b503-6c3941f442c8","Type":"ContainerDied","Data":"465aac688215ab87897b4179de77e3eeb161d08751228e042b0cba27dffa5775"}
Jul 27 22:38:04.303798 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:04.303763    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:04.440026 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:04.439931    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca\") pod \"77c194f6-471b-43e5-b503-6c3941f442c8\" (UID: \"77c194f6-471b-43e5-b503-6c3941f442c8\") "
Jul 27 22:38:04.440288 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:04.440265    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77c194f6-471b-43e5-b503-6c3941f442c8" (UID: "77c194f6-471b-43e5-b503-6c3941f442c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:04.541001 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:04.540962    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77c194f6-471b-43e5-b503-6c3941f442c8-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:05.169570 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:05.169466    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"
Jul 27 22:38:05.169570 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:05.169465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g" event={"ID":"77c194f6-471b-43e5-b503-6c3941f442c8","Type":"ContainerDied","Data":"2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69"}
Jul 27 22:38:05.169785 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:05.169592    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2750aaf13207a0ad97b7a7aaef0679e712fdaf979d1856b2dc6958465d3f7b69"
Jul 27 22:38:06.385041 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.384966    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:38:06.461016 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.460981    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461036    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461057    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461084    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461204 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461181    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nxzg\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461213    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461246    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461262    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy\") pod \"3d7d5d50-8686-47e4-9861-5c42356ab49d\" (UID: \"3d7d5d50-8686-47e4-9861-5c42356ab49d\") "
Jul 27 22:38:06.461481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461291    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:06.461481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461361    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:06.461755 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461510    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:06.461755 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461645    2560 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-certs\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.461755 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461666    2560 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-credential-socket\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.461755 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461674    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:06.461755 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461684    2560 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-workload-socket\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.461962 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.461925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:06.463716 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.463687    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg" (OuterVolumeSpecName: "kube-api-access-2nxzg") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "kube-api-access-2nxzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:06.463849 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.463732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 22:38:06.464039 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.464012    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 22:38:06.464142 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.464051    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d7d5d50-8686-47e4-9861-5c42356ab49d" (UID: "3d7d5d50-8686-47e4-9861-5c42356ab49d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 22:38:06.562892 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562850    2560 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-data\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.562892 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562881    2560 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-podinfo\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.562892 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562890    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxzg\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-kube-api-access-2nxzg\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.562892 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562901    2560 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7d5d50-8686-47e4-9861-5c42356ab49d-istiod-ca-cert\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.563238 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562909    2560 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-token\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:06.563238 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:06.562919    2560 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7d5d50-8686-47e4-9861-5c42356ab49d-istio-envoy\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:07.182169 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.182066    2560 generic.go:356] "Generic (PLEG): container finished" podID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerID="bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f" exitCode=0
Jul 27 22:38:07.182169 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.182106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerDied","Data":"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"}
Jul 27 22:38:07.182169 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.182118    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"
Jul 27 22:38:07.182169 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.182156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z" event={"ID":"3d7d5d50-8686-47e4-9861-5c42356ab49d","Type":"ContainerDied","Data":"601d8ab66d9b4a43749f6c1cdc97a89d2cb927542b5477b91e85203e7133212c"}
Jul 27 22:38:07.182503 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.182181    2560 scope.go:117] "RemoveContainer" containerID="bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"
Jul 27 22:38:07.191905 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.191879    2560 scope.go:117] "RemoveContainer" containerID="a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"
Jul 27 22:38:07.201291 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.201250    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:38:07.202010 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.201992    2560 scope.go:117] "RemoveContainer" containerID="bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"
Jul 27 22:38:07.202349 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:38:07.202325    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f\": container with ID starting with bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f not found: ID does not exist" containerID="bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"
Jul 27 22:38:07.202400 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.202361    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f"} err="failed to get container status \"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f\": rpc error: code = NotFound desc = could not find container \"bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f\": container with ID starting with bd58a18d28fd4b5e29b3568c5dcaa9669cfff26c93f7e6b1ab7be07f26feda6f not found: ID does not exist"
Jul 27 22:38:07.202400 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.202381    2560 scope.go:117] "RemoveContainer" containerID="a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"
Jul 27 22:38:07.202653 ip-10-0-143-110 kubenswrapper[2560]: E0727 22:38:07.202634    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d\": container with ID starting with a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d not found: ID does not exist" containerID="a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"
Jul 27 22:38:07.202702 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.202657    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d"} err="failed to get container status \"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d\": rpc error: code = NotFound desc = could not find container \"a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d\": container with ID starting with a5db4bca2b8b04144ad06d492e029ea880c1755cd1d462572f5d6f000c49f96d not found: ID does not exist"
Jul 27 22:38:07.205982 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:07.205953    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-adca75-openshift-default-c59bc64fd-27s6z"]
Jul 27 22:38:08.915529 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:08.915486    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" path="/var/lib/kubelet/pods/3d7d5d50-8686-47e4-9861-5c42356ab49d/volumes"
Jul 27 22:38:12.014150 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:12.014101    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"]
Jul 27 22:38:12.016758 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:12.016728    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6138cc-29753190-j7f2b"]
Jul 27 22:38:12.916980 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:12.916942    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="643e575d-5e4c-4ccc-bb4a-9467a2506ae1" path="/var/lib/kubelet/pods/643e575d-5e4c-4ccc-bb4a-9467a2506ae1/volumes"
Jul 27 22:38:15.559481 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:15.559427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:38:16.622365 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:16.622327    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"]
Jul 27 22:38:16.625389 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:16.625348    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-6138cc-8lz4g"]
Jul 27 22:38:16.916364 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:16.916257    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c194f6-471b-43e5-b503-6c3941f442c8" path="/var/lib/kubelet/pods/77c194f6-471b-43e5-b503-6c3941f442c8/volumes"
Jul 27 22:38:23.614400 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614363    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"]
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614763    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614775    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614795    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614800    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614814    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c194f6-471b-43e5-b503-6c3941f442c8" containerName="revoke-keys"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614820    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c194f6-471b-43e5-b503-6c3941f442c8" containerName="revoke-keys"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614879    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c194f6-471b-43e5-b503-6c3941f442c8" containerName="revoke-keys"
Jul 27 22:38:23.614938 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.614889    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:38:23.618107 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.618086    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:23.620834 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.620805    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 27 22:38:23.620984 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.620850    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7k4q5\""
Jul 27 22:38:23.625884 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.625846    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"]
Jul 27 22:38:23.630340 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.630306    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb\" (UID: \"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:23.730901 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.730852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb\" (UID: \"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:23.731581 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.731554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb\" (UID: \"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:23.930358 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:23.930264    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:24.074527 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:24.074497    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"]
Jul 27 22:38:24.077208 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:38:24.077172    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62d0dd_7a32_4a52_a9ee_5c0e32fa4975.slice/crio-4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666 WatchSource:0}: Error finding container 4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666: Status 404 returned error can't find the container with id 4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666
Jul 27 22:38:24.274826 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:24.274738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb" event={"ID":"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975","Type":"ContainerStarted","Data":"4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666"}
Jul 27 22:38:25.282617 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:25.282515    2560 generic.go:356] "Generic (PLEG): container finished" podID="be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" containerID="7687e1fe0fc70d915eef8115456b8906d29b67768feaf1533485e2d9853ff7ca" exitCode=0
Jul 27 22:38:25.282617 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:25.282604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb" event={"ID":"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975","Type":"ContainerDied","Data":"7687e1fe0fc70d915eef8115456b8906d29b67768feaf1533485e2d9853ff7ca"}
Jul 27 22:38:26.423314 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:26.423286    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:26.454347 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:26.454308    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca\") pod \"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975\" (UID: \"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975\") "
Jul 27 22:38:26.454712 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:26.454690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" (UID: "be62d0dd-7a32-4a52-a9ee-5c0e32fa4975"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 22:38:26.555771 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:26.555735    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975-maas-api-service-ca\") on node \"ip-10-0-143-110.ec2.internal\" DevicePath \"\""
Jul 27 22:38:27.296003 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:27.295900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb" event={"ID":"be62d0dd-7a32-4a52-a9ee-5c0e32fa4975","Type":"ContainerDied","Data":"4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666"}
Jul 27 22:38:27.296003 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:27.295954    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4981f5fe5c07467db3cd7cf22b814255f94e4137232d5d13a7676b9220ad1666"
Jul 27 22:38:27.296003 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:27.295917    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"
Jul 27 22:38:34.130574 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:34.130525    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"]
Jul 27 22:38:34.136191 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:34.136158    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-0c79f1-29753190-p7lnt"]
Jul 27 22:38:34.916223 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:34.916187    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386dc151-e376-4fd0-9c98-78537aea189b" path="/var/lib/kubelet/pods/386dc151-e376-4fd0-9c98-78537aea189b/volumes"
Jul 27 22:38:35.316170 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:35.316131    2560 scope.go:117] "RemoveContainer" containerID="f14e70ceee2bc9412d2be1d34e9f283f1ac2fe329bd19041ef8f964a645936f9"
Jul 27 22:38:35.325349 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:35.325329    2560 scope.go:117] "RemoveContainer" containerID="726b6040cd705ba25284bddb3ca5286070f9879aa2e7fb35e9e0beba5c9b60ed"
Jul 27 22:38:36.867833 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:36.867793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-zqts9"]
Jul 27 22:38:38.689162 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:38.689121    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"]
Jul 27 22:38:38.693206 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:38.693169    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-0c79f1-fzkjb"]
Jul 27 22:38:38.916047 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:38.916007    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" path="/var/lib/kubelet/pods/be62d0dd-7a32-4a52-a9ee-5c0e32fa4975/volumes"
Jul 27 22:38:52.534998 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:52.534968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2_43afca92-4a6e-4d8b-8185-1042836ea25e/copy-manifests/0.log"
Jul 27 22:38:52.541925 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:52.541887    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2_43afca92-4a6e-4d8b-8185-1042836ea25e/manager/0.log"
Jul 27 22:38:53.506570 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:53.506537    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753190-hnpd4_9d37d472-25a1-4aba-9d05-297891d8713e/cleanup/0.log"
Jul 27 22:38:55.692266 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:55.692233    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g9kpl_9dcbc993-b181-42b4-a342-c2bda2bca1ce/manager/0.log"
Jul 27 22:38:55.815052 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:55.815023    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-zqts9_d86bdeaa-35c4-4785-99dd-da371ff88db6/limitador/0.log"
Jul 27 22:38:56.716090 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:56.716059    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-xt64t_6aa1ed78-4588-47b9-9f87-56f833de56bd/kube-auth-proxy/0.log"
Jul 27 22:38:58.206758 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:38:58.206724    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-q7r5l_2058dc0a-fc13-46f6-b4bc-7ab1c53a17a4/main/0.log"
Jul 27 22:39:03.882743 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.882703    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhm4t/must-gather-hf4qx"]
Jul 27 22:39:03.883142 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.883107    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" containerName="revoke-keys"
Jul 27 22:39:03.883142 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.883120    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" containerName="revoke-keys"
Jul 27 22:39:03.883218 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.883179    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="be62d0dd-7a32-4a52-a9ee-5c0e32fa4975" containerName="revoke-keys"
Jul 27 22:39:03.883218 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.883190    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7d5d50-8686-47e4-9861-5c42356ab49d" containerName="istio-proxy"
Jul 27 22:39:03.886671 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.886645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:03.889221 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.889195    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"openshift-service-ca.crt\""
Jul 27 22:39:03.889692 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.889670    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dhm4t\"/\"kube-root-ca.crt\""
Jul 27 22:39:03.889773 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.889697    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dhm4t\"/\"default-dockercfg-v997v\""
Jul 27 22:39:03.902635 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.902599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/must-gather-hf4qx"]
Jul 27 22:39:03.915607 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.915569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls8lq\" (UniqueName: \"kubernetes.io/projected/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-kube-api-access-ls8lq\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:03.915772 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:03.915665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-must-gather-output\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.016329 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.016285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-must-gather-output\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.016650 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.016347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls8lq\" (UniqueName: \"kubernetes.io/projected/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-kube-api-access-ls8lq\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.016708 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.016661    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-must-gather-output\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.025761 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.025731    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls8lq\" (UniqueName: \"kubernetes.io/projected/e4ef2f15-4b0f-472e-b735-636b3f3d9d4d-kube-api-access-ls8lq\") pod \"must-gather-hf4qx\" (UID: \"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d\") " pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.197295 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.197200    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/must-gather-hf4qx"
Jul 27 22:39:04.336784 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.336747    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/must-gather-hf4qx"]
Jul 27 22:39:04.339404 ip-10-0-143-110 kubenswrapper[2560]: W0727 22:39:04.339366    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ef2f15_4b0f_472e_b735_636b3f3d9d4d.slice/crio-315e36c9dfadb5aad966372d9831425da84f0c35c28c14c8b5152e622faf65ee WatchSource:0}: Error finding container 315e36c9dfadb5aad966372d9831425da84f0c35c28c14c8b5152e622faf65ee: Status 404 returned error can't find the container with id 315e36c9dfadb5aad966372d9831425da84f0c35c28c14c8b5152e622faf65ee
Jul 27 22:39:04.341322 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.341305    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 22:39:04.497404 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:04.497318    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/must-gather-hf4qx" event={"ID":"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d","Type":"ContainerStarted","Data":"315e36c9dfadb5aad966372d9831425da84f0c35c28c14c8b5152e622faf65ee"}
Jul 27 22:39:05.507951 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:05.507900    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/must-gather-hf4qx" event={"ID":"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d","Type":"ContainerStarted","Data":"e3e08f4b6294191e0d71fd88726327b36b801482305d0b21417bb7a8752ffb6f"}
Jul 27 22:39:05.508437 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:05.507968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/must-gather-hf4qx" event={"ID":"e4ef2f15-4b0f-472e-b735-636b3f3d9d4d","Type":"ContainerStarted","Data":"671245ed25697cbc4780db576949132c6b0102676fd519db9abfcc8b9fcb3889"}
Jul 27 22:39:05.528939 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:05.528884    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhm4t/must-gather-hf4qx" podStartSLOduration=1.813865348 podStartE2EDuration="2.528867795s" totalImagesPullingTime="715.002447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 22:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:39:05.527603463 +0000 UTC m=+3211.181032105" watchObservedRunningTime="2026-07-27 22:39:05.528867795 +0000 UTC m=+3211.182296474"
Jul 27 22:39:06.714188 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:06.714157    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7rm99_6fe41b7e-2a93-4659-923b-abd00fdabeec/global-pull-secret-syncer/0.log"
Jul 27 22:39:06.910802 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:06.910767    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dmkp2_c5756508-46af-4918-82a6-c68756753094/konnectivity-agent/0.log"
Jul 27 22:39:07.040013 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:07.039937    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-110.ec2.internal_11d76ec706ed0fd58660b79ceb4322ce/haproxy/0.log"
Jul 27 22:39:12.891993 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:12.891855    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-g9kpl_9dcbc993-b181-42b4-a342-c2bda2bca1ce/manager/0.log"
Jul 27 22:39:12.921665 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:12.921587    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-zqts9_d86bdeaa-35c4-4785-99dd-da371ff88db6/limitador/0.log"
Jul 27 22:39:14.812255 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.812224    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/alertmanager/0.log"
Jul 27 22:39:14.840229 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.840195    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/config-reloader/0.log"
Jul 27 22:39:14.865544 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.865507    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/kube-rbac-proxy-web/0.log"
Jul 27 22:39:14.896270 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.896232    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/kube-rbac-proxy/0.log"
Jul 27 22:39:14.927622 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.927586    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/kube-rbac-proxy-metric/0.log"
Jul 27 22:39:14.952947 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.952910    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/prom-label-proxy/0.log"
Jul 27 22:39:14.979195 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:14.979158    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d83fa88e-3a53-4fb8-8d48-1e4bc1a60ad4/init-config-reloader/0.log"
Jul 27 22:39:15.068595 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.068499    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-8cxmp_3faccd65-c981-4247-971e-1e39178a6710/kube-state-metrics/0.log"
Jul 27 22:39:15.087962 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.087909    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-8cxmp_3faccd65-c981-4247-971e-1e39178a6710/kube-rbac-proxy-main/0.log"
Jul 27 22:39:15.114666 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.114633    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-8cxmp_3faccd65-c981-4247-971e-1e39178a6710/kube-rbac-proxy-self/0.log"
Jul 27 22:39:15.164710 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.164670    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7b669bc878-mxn88_897af678-0db9-4f38-816e-2bd664b91361/metrics-server/0.log"
Jul 27 22:39:15.199027 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.198992    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-f5mzs_4d8ad193-3e0b-4f14-a17c-685c9c034e1b/monitoring-plugin/0.log"
Jul 27 22:39:15.232165 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.232128    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nt2p_670d8499-96e3-452a-8545-8f8bd041e369/node-exporter/0.log"
Jul 27 22:39:15.259133 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.259097    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nt2p_670d8499-96e3-452a-8545-8f8bd041e369/kube-rbac-proxy/0.log"
Jul 27 22:39:15.285540 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.285504    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8nt2p_670d8499-96e3-452a-8545-8f8bd041e369/init-textfile/0.log"
Jul 27 22:39:15.495216 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.495182    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ml84k_41e7455e-cd15-4f81-97bc-d8a0b6eab446/kube-rbac-proxy-main/0.log"
Jul 27 22:39:15.521067 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.521006    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ml84k_41e7455e-cd15-4f81-97bc-d8a0b6eab446/kube-rbac-proxy-self/0.log"
Jul 27 22:39:15.555837 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.555808    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-ml84k_41e7455e-cd15-4f81-97bc-d8a0b6eab446/openshift-state-metrics/0.log"
Jul 27 22:39:15.596379 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.596347    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/prometheus/0.log"
Jul 27 22:39:15.619698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.619660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/config-reloader/0.log"
Jul 27 22:39:15.643411 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.643380    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/thanos-sidecar/0.log"
Jul 27 22:39:15.667044 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.667013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/kube-rbac-proxy-web/0.log"
Jul 27 22:39:15.691210 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.691178    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/kube-rbac-proxy/0.log"
Jul 27 22:39:15.731905 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.731823    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/kube-rbac-proxy-thanos/0.log"
Jul 27 22:39:15.758758 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.758723    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_463d472c-cc06-4e60-8924-8e757d17991a/init-config-reloader/0.log"
Jul 27 22:39:15.792411 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.792366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-8n6vb_203743aa-5c3d-48a9-a4da-19ef0e94bc20/prometheus-operator/0.log"
Jul 27 22:39:15.817979 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.817945    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-8n6vb_203743aa-5c3d-48a9-a4da-19ef0e94bc20/kube-rbac-proxy/0.log"
Jul 27 22:39:15.846503 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.846467    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-m9mqs_7f7d3091-25db-4808-ad26-da8555fdccf3/prometheus-operator-admission-webhook/0.log"
Jul 27 22:39:15.882546 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.882495    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5586cd6d9b-5qfqs_7dc9b9ca-246a-434f-9756-bfd43de6df05/telemeter-client/0.log"
Jul 27 22:39:15.914241 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.914211    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5586cd6d9b-5qfqs_7dc9b9ca-246a-434f-9756-bfd43de6df05/reload/0.log"
Jul 27 22:39:15.937171 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.937142    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5586cd6d9b-5qfqs_7dc9b9ca-246a-434f-9756-bfd43de6df05/kube-rbac-proxy/0.log"
Jul 27 22:39:15.972879 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.972849    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/thanos-query/0.log"
Jul 27 22:39:15.996694 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:15.996604    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/kube-rbac-proxy-web/0.log"
Jul 27 22:39:16.021859 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:16.021833    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/kube-rbac-proxy/0.log"
Jul 27 22:39:16.050215 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:16.050184    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/prom-label-proxy/0.log"
Jul 27 22:39:16.081357 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:16.081322    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/kube-rbac-proxy-rules/0.log"
Jul 27 22:39:16.110599 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:16.110571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6ddd7657b4-qcvcq_b3e4b5db-eba5-4147-8a8b-6a3badceea0d/kube-rbac-proxy-metrics/0.log"
Jul 27 22:39:17.322822 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.322778    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"]
Jul 27 22:39:17.331909 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.331861    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.345056 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.345020    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"]
Jul 27 22:39:17.479555 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.479516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-lib-modules\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.479555 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.479561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pclgh\" (UniqueName: \"kubernetes.io/projected/38257ff3-9bde-4192-b80b-fdae0f661a2e-kube-api-access-pclgh\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.479825 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.479595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-sys\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.479825 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.479613    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-podres\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.479825 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.479707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-proc\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581404 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581294    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-lib-modules\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581404 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pclgh\" (UniqueName: \"kubernetes.io/projected/38257ff3-9bde-4192-b80b-fdae0f661a2e-kube-api-access-pclgh\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581404 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-sys\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581424    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-podres\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-proc\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581745 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581677    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-proc\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.581902 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.581810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-lib-modules\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.582177 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.582149    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-sys\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.582348 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.582315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/38257ff3-9bde-4192-b80b-fdae0f661a2e-podres\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.591011 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.590979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pclgh\" (UniqueName: \"kubernetes.io/projected/38257ff3-9bde-4192-b80b-fdae0f661a2e-kube-api-access-pclgh\") pod \"perf-node-gather-daemonset-7f5fh\" (UID: \"38257ff3-9bde-4192-b80b-fdae0f661a2e\") " pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.647276 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.647227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:17.874982 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:17.874639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"]
Jul 27 22:39:18.616603 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:18.616562    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh" event={"ID":"38257ff3-9bde-4192-b80b-fdae0f661a2e","Type":"ContainerStarted","Data":"0ea90d2fa85ee608c060abf925aa6b14f9eeaf3782b656133ac9b3d9028f8629"}
Jul 27 22:39:18.617101 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:18.616632    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh" event={"ID":"38257ff3-9bde-4192-b80b-fdae0f661a2e","Type":"ContainerStarted","Data":"01594a42298e9bf1e723ae4df233214647dc67c94afefa1b07f53a56083f7208"}
Jul 27 22:39:18.617101 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:18.616657    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:18.636154 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:18.636093    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh" podStartSLOduration=1.6360715 podStartE2EDuration="1.6360715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 22:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 22:39:18.634730754 +0000 UTC m=+3224.288159396" watchObservedRunningTime="2026-07-27 22:39:18.6360715 +0000 UTC m=+3224.289500142"
Jul 27 22:39:18.680927 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:18.680850    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b8dff4fb6-hj7jd_b73db49d-c709-4fa2-821d-6ac1196923cf/console/0.log"
Jul 27 22:39:20.416256 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:20.416226    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m9h8_52d4e105-b378-4b64-8540-fb84c7d86ac9/dns/0.log"
Jul 27 22:39:20.440139 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:20.440110    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8m9h8_52d4e105-b378-4b64-8540-fb84c7d86ac9/kube-rbac-proxy/0.log"
Jul 27 22:39:20.621285 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:20.621252    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lfrh2_6ac976c8-57d2-4848-8d3b-1b331e5eb3b2/dns-node-resolver/0.log"
Jul 27 22:39:21.279310 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:21.279282    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ts9l5_70ec5f21-87aa-453b-8b4b-999b7484f0b1/node-ca/0.log"
Jul 27 22:39:22.581708 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:22.581678    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-xt64t_6aa1ed78-4588-47b9-9f87-56f833de56bd/kube-auth-proxy/0.log"
Jul 27 22:39:23.655113 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:23.655082    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tj4z9_7d4cbc3c-bf7c-4dbd-94f2-3e9e3caef4ca/serve-healthcheck-canary/0.log"
Jul 27 22:39:24.227586 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:24.227520    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qm5vp_71c0cb96-1bc0-411c-88bf-2137aae6a264/kube-rbac-proxy/0.log"
Jul 27 22:39:24.253111 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:24.253080    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qm5vp_71c0cb96-1bc0-411c-88bf-2137aae6a264/exporter/0.log"
Jul 27 22:39:24.277689 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:24.277656    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qm5vp_71c0cb96-1bc0-411c-88bf-2137aae6a264/extractor/0.log"
Jul 27 22:39:24.637986 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:24.637957    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dhm4t/perf-node-gather-daemonset-7f5fh"
Jul 27 22:39:27.763301 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:27.763215    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2_43afca92-4a6e-4d8b-8185-1042836ea25e/manager/0.log"
Jul 27 22:39:27.788141 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:27.788111    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7f6d4bf4d4-g2xr2_43afca92-4a6e-4d8b-8185-1042836ea25e/copy-manifests/0.log"
Jul 27 22:39:28.649698 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:28.649668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-xt64t_6aa1ed78-4588-47b9-9f87-56f833de56bd/kube-auth-proxy/0.log"
Jul 27 22:39:29.396297 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:29.396208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-647fb87ffb-4hjcg_e21ffd1a-601c-4740-b5df-5858b00e2df4/manager/0.log"
Jul 27 22:39:30.547883 ip-10-0-143-110 kubenswrapper[2560]: I0727 22:39:30.547854    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5794464b4b-xt64t_6aa1ed78-4588-47b9-9f87-56f833de56bd/kube-auth-proxy/0.log"